|
|
Post by bluefish on Jul 19, 2021 20:04:45 GMT
I'm a grouchy old sysadmin devops person who's been spoiled by cloudy ways of doing things. I'm now trying to drag some of those automation patterns to my current project which involves physical servers (the horror!) The task at hand involves cluster building, which requires that a bunch of servers are able to sync up with one another upon their first boot after OS installation. I recognize that I could bake an SSH trusted CA into the OS image, impose the requisite configuration elements from a central automation point (ansible/saltstack/puppet/chef), but I'd rather not have a centralized long-lived automation hub. Telegraphing a secret through the BMC, into the host OS is a better fit here. Using that secret, the OS can query the parameters it needs from a central password vault, etc... So... Do the commonly available Redfish implementations (I'm using Dell boxes, but would prefer a non-proprietary solution) have the capabilities I'd need to make this work? It looks to me like there are two main requirements here: - A place to store data within the BMC - something like a key/value store would be ideal. Enough space to keep an AWS credential or Hashicorp Vault token.
- Unauthenticated access to that data from within the host OS
For (1), there seem to be endless places to write strings in the iDRAC I'm playing with, but none jump out at me as a general purpose key/value store.
For (2), I've had a look at Redfish DSP0270, could use some assistance in parsing the details.
The examples mentioned in DSP0270 are: - A USB Network Connection between a Host and a Redfish Service
- A Host PCIe NIC that connects to a Manager NIC
- A Host PCIe NIC that connects to a management LAN that connects to a Redfish Service
Does the first one mean I should expect to find an AX88772 or similar within the host OS? I'd expect I'd have noticed it if one were there. Maybe I've neglected to switch it on? The second example also suggests an adapter I'd expect to have noticed. In this case, I'm reading "connects to" to mean "internal directly cabled to the BMC" in the same way that the serial port seems to be directly linked to the BMC. The third example would (in my case anyway) require unauthenticated access to the Redfish API, so that's straight out. Thanks! |
|
|
|
Post by mraineri on Jul 21, 2021 15:01:27 GMT
For (1), are you looking at what is essentially an arbitrary way for a user to leave data on a Redfish service? Do you have some other examples of the types of things you'd want to leave behind? Things like tokens or keys might be seen as problematic from a security perspective, but if you have other pieces of data we might be able to discuss standardizing on a place to put some information.
For (2), essentially the DMTF has definitions for how a local host can perform management via Redfish with its local manager. DSP0270 defines the constructs of a "host interface". Effectively managers will emulate some sort of NIC for the host (either as a USB device or a PCIe device), and this creates an internal point-to-point network between the host and its manager; no cables needed, but I would expect some sort of NIC type of device to be enumerated (similar to an AX88772, but this really depends on how the manager virtualizes the device). The examples in the list are just possible approaches a system vendor might use for implementing the host interface, but the most common approach I've seen has been using the internal USB path between the host and manager. At the end of the day though, if a vendor is requiring you to plug in new devices to get this connectivity, that's a bad design in my opinion; the intent is for it to all be self-contained so a user does not need additional hardware or cabling.
Assuming the interface is enabled, you can use a tool like this (https://github.com/nhorman/redfish-finder) to find the entry in the SMBIOS record for your interface, and it will update /etc/hosts to point "redfish-localhost" to the IP address of your local manager.
It is possible to use this host interface in an unauthenticated manner, but I generally don't see that enabled by default. The HostInterface resource in the model contains an "AuthenticationModes" property to control the authentication methods allowed on the interface. You should be able to add the value "AuthNone" to allow for unauthenticated access.
|
|
|
|
Post by bluefish on Jul 23, 2021 15:44:11 GMT
Things like tokens or keys might be seen as problematic from a security perspective For (1), I really do want to use this capability for secrets, to solve the general problem of bringing newly-racked, OS not-yet-installed systems under management. The problem here is that the new system doesn't have any way to authenticate itself to network services, so how can it be brought online safely? For purposes of this discussion, I'm assuming a secure channel is already available between the automation system and the BMC. Possibilites: - Transport a credential between the automation system and the new OS via the serial port (awful, but I can make this work, and it's a secure channel)
- Introduce some authentication hardware like a yubikey (easy, plug it in) or TPM (tricker to load without help from the BMC, I imagine)
- Do initial boot from secure physical media (a USB stick) preloaded with secrets
- Something else? I'd happily entertain a "do it this other way" discussion
I don't really see the problem that comes from entrusting the BMC with a credential used to identify the server ... I mean, we already trust it to control the server's keyboard, boot media, power switch, etc... Are we really crossing a line here? If I'm following you correctly on (2), I think it boils down to: - A "hardware NIC" internally wired to the BMC. This may need to be explicitly enabled within the BMC.
- DSP0270 exposes the relevant configuration information required for the OS to make use of that NIC (IP address and whatnot)
So far so good?
When you mention "the USB path between the host and manager", are we talking about something that looks like an Ethernet controller on the host's USB bus, or is this an alternative to that model? Redfish-finder will "find the entry in the SMBIOS" and "update /etc/hosts"... Who is responsible for configuring (`ip addr add <address> dev <interface>`, etc...) the IP-speaking interface within the OS in this scenario? It sounds like I've just not yet managed to find "enable host-facing device" lever in my iDRAC menus, because the `ip link list` output of my Ubuntu/Dell boxes (I'm using R6525 and R6515) only mentions interfaces associated with physical connectors on the back panel (plus loopback), and the Linux installer didn't mention it as an interface I might attempt to use for installation. I understand why you'd have reservations about that, of course. The ideal scenario would include some role-based capability (the OS doesn't need unauthenticated access to all of the BMC's powers), and use of a short-lived or one-time-use credential (the breadcrumb) stored in the BMC (not the BMC's problem to solve, obviously) Finally, thank you for your reply! I came here after hearing you mention this forum on a podcast, was hoping you'd turn up to help me. |
|
|
|
Post by mraineri on Jul 28, 2021 13:12:10 GMT
For #1, that might be something you can manage with virtual media presented by the BMC. If you can host an ISO or other type of media on a system, you can use the virtual media resources to direct your manager to mount that media and present it to the host system as a CD or other mountable device; the host will just see this as a local media source and not need to go out on the network to access the data. I'll ask others in the forum to see if they have other opinions on this. At least with this type of approach, you should be able to see fairly wide support for the capability already.
For #2, you're right in that there is some local hardware interface between the host and BMC with a NIC, and on most systems I've seen this is going to look like a USB attached Ethernet controller. IP address assignment is typically performed by the BMC; the BMC has a local DHCP server to give an address to the host when it comes up. So, if everything works well and is enabled properly, you shouldn't need to configure anything on the OS side of things. I have access to a few systems and can try to get a quick workflow together if that will help. At the same time though, if my suggestion for #1 pans out, you likely would not need to use the host interface since the data would be presented as a local media source.
|
|
|
|
Post by bluefish on Jul 28, 2021 22:41:01 GMT
Virtual media is a great idea. Thank you for the suggestion. ... some local hardware interface between the host and BMC with a NIC "with a NIC" <- thank you for clarifying. Your prior comment left me wondering if it could be some other type of USB interface (though it didn't seem likely). I can only imagine that the "hardware" is either disabled, or so exotic that I don't have a driver, because I don't see it with `ip link list`, and the installer doesn't suggest pulling packages via that interface, etc... Ooh, that's interesting. I'm curious to have a look at the DHCP transaction and the (claimed) speed of the NIC, as these things impact host behavior in interesting ways on various platforms. Reminds me of Lan Turtle / Bash Bunny. #1 was a great idea, and that's likely what I'll do. My automation can spit out and attach an ISO easily enough. Though I am curious to play with the host interface, assuming my R6525 or R6515 have it. Thanks again! |
|
