147 pointsby mooredsa year ago11 comments
  • PAPPPmAca year ago
    I've been using Warewulf (&co.) for provisioning bare-metal clusters for decades (back into the Perceus days between Warewulf 1 and 2), it's a solid easy-to-comprehend tool that does things in ways that are transparent and built from generic [u/li]nux tools enough that they're not hard to think about when needed, but automated enough you usually don't have to.

    Definitely shows its research roots, best-tested with RHEL-alikes, reasonably well tested with Suse and Debian, and you may be in for some extra work if you need provision something else, but that pretty much covers the common cases (and it integrates with containerization tools if you need some specific environment on the nodes).

    It's a nice to have when you need to spin many nodes.

    • superb_deva year ago
      It’s that old? I can’t believe it took me this long to find Warewulf! I’ve tried the more complex solutions and this looks like what I’ve always dreamed of
    • generalizationsa year ago
      Why use warewulf? Seems like there have got to be simpler solutions for dealing with bare metal clusters than all of this.
      • anderbubblea year ago
        Perhaps you have not seen some of the other solutions out there in this space.

        Warewulf _is_ the simpler solution.

      • mikestorrenta year ago
        Always feels like it will be simpler... you start with some iPXE, start building, and 6 months later you have a poor imitation of a product like this that works only for your specific use cases and causes you a headache if the company pivots and you have to make it do something new.

        Been there, built that. Next time I'm using something with a community, and if it doesn't do what I need, I'm contributing upstream until it does.

        • generalizationsa year ago
          Ooof sounds like experience talking. I guess flexibility is kinda paramount here. Do you think you'd use something like warewulf then?
  • MortyWavesa year ago
    Where does this fit in the Ansible + PXE boot vs Terraform vs NixOS scale? Seems to be within that space, but before the "infrastructure as code" phrase was coined.
    • anderbubblea year ago
      It's PXE boot for mostly stateless node / disk images, with a template-based overlay system for customizing the image as it's applied to the node.
      • generalizationsa year ago
        So with the overlay you can make node-specific changes? I was reading through the initial setup guide but I couldn't figure out how you actually specify which node gets which overlays (or one-off edits of whatever kind).
  • mooredsa year ago
  • 4ggr0a year ago
    Right now I'm searching for the optimal solution to host containers in a small but cool way. Thinking about just using a plain Linux host, configured via ansible and compose.

    Used Portainer so far, but that's a bit bloated for my simple use (one host, no-HA, lab env). Kubernetes is way too complex as well.

    Warewulf sounds fun to try :D all of my profiles would probably only have one node. Does Warewulf make a fraction of sense when having a tiny, quasi-local environment?

    EDIT: ah, nevermind. stateless and temporary makes no sense for my usecase, as my containers will run 24/7 with rare changes. But I will think about Warewulf if I ever dive into large-scale containerization :)

    • huijzera year ago
      If you want small and cool, Wasmtime should be mentioned. It’s a WebAssembly runtime to it can only run WebAssembly programs, which narrows the set of programs that can be executed. However, if you can get your program in WebAssembly then Wasmtime is very cool. Startup time is a few ms versus hundreds of ms for Docker. Memory usage is also orders of magnitude less. And security guarantees are similar.
    • proxysnaa year ago
      > plain Linux host, configured via ansible and compose.

      There is your answer. I'd would also add watchtower to keep your containers up to date.

      • 4ggr0a year ago
        great suggestion! thanks, will definitely implement this :)
    • udev4096a year ago
      If you really want a lightweight experience, go with alpine and then run all the daemon processes using supervisor [1] and for HA, you can use keepalived which uses VRRP for HA

      [1] - https://github.com/Supervisor/supervisor

    • brazzya year ago
      > Thinking about just using a plain Linux host, configured via ansible and compose.

      That's what I've been using since 2019 (plus caddy as reverse proxy for various web services), and in general I've been happy. Upgrading to docker-conpose-v2 caused me some headaches recently though, and I'll soon have to upgrade the underlying Ubuntu server, which I am dreading.

    • roygbiv2a year ago
      Nomad is very cool for hosting containers. Not nearly as complex as kunernetes and you can scale to multiple hosts if needed easily.
    • omnimusa year ago
      Maybe Dockge? Its like Portainer light/cool.
      • 4ggr0a year ago
        Also great suggestion, this looks like a nice addition to the stack! thx :)
    • throawayonthea year ago
      [dead]
  • mkespera year ago
    Weird this does not make use of IPv6. I'd thought this was a given if you have tens of thousands of nodes.
    • anderbubblea year ago
      We're always looking towards IPv6 support! And you _can_ do it today, with a little bit of work. But it's been difficult to prioritize in the main project when so few of our users (read: maybe one) have expressed interest.
      • withinboredoma year ago
        > when so few of our users

        existing users, I suppose. New users are looking for exactly this feature and will walk away. So, you can now count it up to 3.

        IPv4 has gotten quite expensive. A newer company I'm working with doesn't even have IPv4 access past the edge. There is just a little proxy that handles IPv4 translation on the edge; it barely gets any traffic.

    • 8organicbitsa year ago
      10.0.0.0/16 supports 64k hosts, so it seems it would fit in IPv4.
      • anderbubblea year ago
        That's the thing: people don't put their HPC clusters on the public Internet; so an internal IPv4 network just keeps being fine.
        • ale42a year ago
          True, but on the other hand it might be that all private use blocks are already in use (10.0.0.0/8 is totally in use in our internal LAN), so if I want the nodes to reach those private IPs, I can't assign the same block. And we do have services on IPv6.
    • wmfa year ago
      It's from 2001 when IPv6 didn't really exist. I love IPv6 but it's basically crackpot retrocomputing tech at this point.
      • generalizationsa year ago
        Can't decide if the retrocomputing you mean is the warewulf or the ipv6.
        • wmfa year ago
          IPv6 is retrocomputing. I don't know much about Warewulf.
          • generalizationsa year ago
            Huh. Did not realize we'd given up on that. Did the industry just settle on IPv4 after all?
            • mcpherrinma year ago
              Really depends on who you ask. You still need v4 to be "globally reachable", but v6 is optional.

              AWS seems to finally be feeling the pinch of IPv4 exhaustion and is pushing v6 support everywhere now, and starting to charge for v4.

              Mobile networks already have, and many are natively IPv6, with NAT64/464XLAT or other tech for bridging to v4. Apple's App store requires apps to support IPv6-only networks.

              CDNs and clouds etc mean that websites don't even really need to worry about their own IP allocation, and just let their provider figure out exposing things worldwide.

              • joshstrangea year ago
                > Apple's App store requires apps to support IPv6-only networks.

                I read that and thought "huh, is that recent?" and found posts that were 9 years old about it. I guess apps just have to work on an IPv6-only network but I'm honestly surprised my apps do. I don't test in IPv6, my home network has it disabled, most of my servers don't have anything for IPv6 that I know of. Odd.

                • ianburrella year ago
                  If you use the official network API, don't go out of way to use the old one, any app should support IPv6.
                  • Arnaviona year ago
                    As long as the the servers they talk to have v6 addresses. Or does Apple implement some DNS64+NAT64 / 464XLAT setup transparently in its OSes too?
                    • mcpherrinma year ago
                      I think Apple tests on "IPv6 Only" networks that have DNS64/NAT64 to reach v4 servers
                    • jcgla year ago
                      They do implement a CLAT, both in iOS and macOS afaiu.
              • aitchnyua year ago
                Tangential, will email host spam reputation rely on IPv4 scarcity forever?
                • dspilletta year ago
                  For longer, I expect. For a long time email has been partially centralised so for most real people and a lot of systems mail goes out through a specific host (or small number of hosts) on the edge of their network or completely outside it (sending individuals sending via services like gmail, and systems using services like sendgrid, and so forth) so the need to push for IPv6 is less apparent for mail sending than a number of other things. There are orders of magnitude less hosts sending mail than, say, making HTTP(S) requests.
  • yankoa year ago
    https://www.rocksclusters.org/#

    One more good project from the very past

  • naikroveka year ago
    > Warewulf is a stateless and diskless container OS provisioning system

    are we just always assuming that everything is Linux, now? There are other operating systems, and this one only supports Linux.

    I know that once you start working in Linux a lot, you start to ignore other operating systems, but they still exist.

    This is not an "OS provisioning system", this is a "Linux provisioning system".

    • linuxftwa year ago
      If we're going to be pedantic, Why not GNU/Linux, as Linux is just the kernel? Of course, not all modern distros are based on the GNU userlands.
    • yjftsjthsd-ha year ago
      Linux is a kernel, not an operating system; it's a system for provisioning Linux-based OSs.
  • mrbluecoata year ago
    Is it just for short-term HPC compute job use cases or also IoT bare metal with OTA upgrades?
    • CaliforniaKarla year ago
      Its main thing is taking a system that’s just powered on, and giving it an operating system. Said system can then run HPC jobs (or shell sessions, or web sites, or data-transfer); until it’s rebooted, and the cycle begins again.

      I don’t know how well it would work in IoT: The device needs to PXE-boot, which requires support from the DHCP server and the hardware boot environment (UEFI).

    • a year ago
      undefined
  • suzzer99a year ago
    Did anyone else read this as "stateless and dickless" at first and wonder what the heck was going on?

    No?

    I'll show myself out.

    • yrega year ago
      I did as well, "iskl" seems like an unsual english char sequence, at least compared to "ickl".
    • a year ago
      undefined
  • sneaka year ago
    This project’s README tells me so little.

    https://sneak.berlin/20241224/readme-howto/

    • sevga year ago
      The words in blue that are underlined are called hyperlinks. You can click them ;)

      This readme howto is truly way too long and excessively prescriptive, and the author goes too far with his inserting his opinions (ie, don’t use github, don’t use discord etc.). I couldn’t possibly recommend this howto.

    • xenophonfa year ago
      The documentation link is _right there_.