Behind the Eyes

Tag: SwitchDEV

  • Deploying SwitchDev with MAAS

    Hello! So in this post, I’m going to show how to deploy a Mellanox switch with the latest firmware installed using Canonical’s Metal-As-A-Service, or MAAS. It’s actually pretty easy since with the latest release, 16.10, it has SwitchDev already enabled. There are just a few little things you need to do to make this deployment a much easier thing.

    The point of this article is so that you can install your Mellanox switch, connect it to your MAAS managed network, and deploy the switch and it will be ready to go.

    First thing you need to do is on the MAAS server, add a tag. This tag will make use of the “kernel_opts” meta tag so that we can use the console on the device. By default, MAAS wants to use tty and ttyS0, which is fine most of the time, but because Mellanox uses Console redirection, we have to include the baud rate and other settings otherwise we get no displayed output. This is easy enough. Log in to your MAAS server, and login to the MAAS CLI.

    NOTE: I like to export my API Key in my home directory so I don’t have to type it. I’ve seen some people make it an env variable so that they can just call it with a $APIKEY variable, either way, you can do this with the following command to make it easier to login:

    sudo maas apikey --username $USER > maas.apikey

    Now, you can login to the MAAS CLI:

    maas login default http://maas/MAAS `cat ~/maas.apikey`

    Now you can create the tag for the switch’s console:

    maas default tags create name='serial_console' comment='115200 serial console' kernel_opts='console=ttyS0,115200n8'

    You can add as many other tags and kernel_opts as necessary for each different switch you want to have managed by MAAS.

    Next, we have to download Ubuntu Yakkety Yak into MAAS if you don’t already have it. To do this, go to the Images tab in the MAAS WebUI, check the 16.10 checkbox and click Save Selection. After a few moments MAAS will download the image and you’ll be able to use it to deploy systems with. After you save selection, you can move on to the next step since by the time you’re done with it, MAAS will have downloaded the image and you’ll be ready to deploy.

    Now, you are ready to have the switch be provisioned by MAAS. Plug in the switch to a managed power supply that MAAS can manage. I use DLI’s PDU in my home lab with great success. Just make sure it is configured properly so that MAAS can use it. If you have any questions about this, leave it in the comments and I’ll reply back.

    Login to the PDU’s web interface and turn on the switch. Make sure you have the console cable connected to either a laptop or a serial concentrator that you can access to watch the device come up. You have to get into the BIOS to change the boot order. By default the Mellanox switch boots to the installed hard drive since it comes with ONIE pre-loaded on it. This will cause the system to never boot via PXE, which is what MAAS needs for it to work. When you get the AMI BIOS screen press CTRL-B, there is no prompt, you just press it and it will take you into the BIOS. Under the Boot tab, move the P0:InnoDisk to the third option and make sure that the Intel GB NIC’s are first and second. Then Save and Exit the BIOS and it will reboot the switch. You will get two PXE BIOS boot messages, one for each NIC, then the BIOS display again, then you get the familiar PXE Boot process. The device will hit MAAS and start the Provisioning. You will get no display on your console, but when it is finished, you will hear the fans over speed and get a red Fault indication on the device. Since MAAS tries to power down the device with the shutdown command, and there is no power off with a switch, it goes into a faulted state. From the PDU interface, shutdown the switch. Go to MAAS and you will see your device there in a NEW status. You need to adjust the power for it, so click on it in the MAAS WebUI and scroll down to Power. Put in the Power information for the device.

    NOTE: If you are using the DLI PDU, you will get no power indication. This is by design. For power status, you will have to look at the PDU web interface or look at the switch and verify it is on.

    Once you have the power setup for the switch, Click on Edit for the tags selection and put in serial_console (It should auto-populate as you type it) and click save. Now you are ready to commission the switch. From the Task pull down, select Commission. The switch will power up and start to boot up. NOTE: You may want to watch this since sometimes I have seen it change the boot order back to hard drive after the faulted state. If it does, just go back into the BIOS and change the boot order back to PXE. After you commission the device, this seems to not happen anymore.

    After commissioning, You will have a Ready state in MAAS. You are now ready to deploy the switch. Click on the switch, and select Deploy from the Task pulldown. You will be given the option of what Operating System and version to install on the device. Select Yakkety Yak and click Deploy. You can watch the device come up, install and deploy. After the reboot, you will have a deployed switch. You can login to the device via SSH with the ubuntu@device-name. I highly recommend adding a local user account with password on the device so that you have a way to get into the device locally if your management network ever goes down and you need to configure the device.

    The last thing you have to do is add udev rules so that the front panel gets the correct names for the device. This is just like with the switchdev in Ubuntu-Core on the switch. First you need the get the switchid. To do that run the following command:

    ip link show dev eth0 | awk '{print $9}'

    The output is the switch id. Note that since you will need it for the udev rule. Now we need to create it. To do that, login to the switch and run:

    sudo vi /etc/udev/rules.d/10_custom.rules

    and enter the following

    SUBSYSTEM=="net", ACTION=="add", ATTR{phys_switch_id}=="<switch-id>", ATTR{phys_port_name}!="", NAME="sw1$attr{phys_port_name}"

    <switch-id> is the value you got from the previous command/step.

    Reboot the switch and verify that the interfaces are named properly:

    ip link show dev sw1p1

    You should get the following:

    34: sw1p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop switchid 7cfe90f02dc0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 7c:fe:90:f0:2d:fd brd ff:ff:ff:ff:ff:ff
    At this point you are ready to setup your device as you see fit using standard Linux commands.

    Please let me know if the comments if you have any questions.

    Thanks!

     

  • SwitchDev in Ubuntu-Core? Yes Please!

    Hello fellow Snappy and Networking enthusiasts. Welcome to my next blog post. This post is mostly to go over building SwitchDev into the Snappy Kernel using the latest kernel. It’s fairly straight forward if you have read my blog entry on how to build a custom kernel snap. I will touch on that a little here as well as go into some things I ran into during the initial build of this.

    First things first, make sure you are running on Ubuntu 16.04 with the latest updates and snapcraft (sudo apt install snapcraft -y), and do the necessary updates:

    sudo apt update && sudo apt upgrade -y

    One thing I did differently that I did in my previous kernel snap post (Success in building a Kernel Snap in snapcraft 2.8.4) is instead of downloading the kernel source from Ubuntu, I got the latest and greatest kernel from Kernel.org, (4.7.0-RC5) but I also had snapcraft download it via git and build. I also didn’t create a kconfigfile like last time, but instead, used the kbuild mechanism to run make defconfig and make oldconfig for me so that it was up to date. I’ll explain how I did this.

    The first thing I did was create a directory to work in called switchdev. mkdir ~/switchdev. I then copied my kernel config from my workstation, and name it 44.config. cp /boot/config-`uname -r` ~/switchdev/44.config

    I then changed my directory to cd ~/switchdev and ran snapcraft init to build the initial snapcraft.yaml file. I then modified the snapcraft.yaml file so it looked like the following:

    name: switchdev-kernel
version: 4.7.0-RC5
summary: SwitchDev Custom Kernel
description: Custom Kernel for Snappy including SwitchDev
type: kernel
confinement: strict
parts:
  kernel:
    plugin: kernel
    source: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
    source-type: git
    kdefconfig: [defconfig, 44.config]
    kconfigs:
      - CONFIG_LOCALVERSION=-snappy"
      - CONFIG_DEBUG_INFO=n
      - CONFIG_SQUASHFS=m
      - CONFIG_NET_SWITCHDEV=y
    kernel-initrd-modules:
      - squashfs
      - ahci

    I then ran snapcraft pull . I ran pull because I have to put my 44.config in the kernel/configs directory so that make oldconfig has something to go against, and I have all the required drivers and modules for a stock Ubuntu kernel.

    By putting my 44.config and using defconfig, the kdefconfig parameter and the kconfigs parameter will be run to create an initial .config. Then the kernel plugin runs "yes" "" | make oldconfig to have an updated .config for building the kernel. So by pulling in all the files, I can then copy 44.config to the correct location:

    cp 44.config parts/kernel/src/kernel/configs/

    I then run snapcraft and grab something to snack on since it will take about an hour to build the kernel snap.

    Once completed, I have a kernel snap named switchdev-kernel_4.7.0-RC5_amd64.snap. I then run this kernel snap through the ubuntu-device-flash application to create a Ubuntu-Core image that I can then install onto a switch. You have to use the ubuntu-device-flash from people.canonical.com/~mvo/all-snaps/ubuntu-device-flash and make it executable (chmod +x ubuntu-device-flash)so that you can run this. You also need kpartx installed (sudo apt install kpartx) on your machine since it uses that to build the image. Once you have all of this, simply run:

    sudo ./ubuntu-device-flash core 16 --channel=edge --os=ubuntu-core --gadget=canonical-pc --kernel=switchdev-kernel_4.7.0-RC5_amd64.snap -o switchdev.img

    After that completes, burn your image onto your switch by either running it through your ONIE installer package creation tool, or by using dd or whatever other method for getting an Operating System on your whitebox switch.

    One thing I noticed once the system came up, was that none of the ports lined up with what the devices were called. Some were called eth0 to eth35, with some missing in between. Some were called renamed7-14, and one was named sw1_phys_port_namex. To fix this so that I could program the switch properly, I had to create a udev rules file. First thing I had to do was get the switchid. To do this, I ran

    ip link show eth8 | grep switchid

    and the value after switchid was what I needed. I then created /etc/udev/rules.d/10_custom.rules and put the following in:

    SUBSYSTEM=="net", ACTION=="add", ATTR{phys_switch_id}=="switchid", ATTR{phys_port_name}!="", NAME="sw1$attr{phys_port_name}"

    I saved the file and then rebooted the switch and when it came up, all the front panel ports were named sw1p1-sw1p32. I could then use the ip command to manage the ports on the switch and even set static routes and move packets around.

    Let me know how it goes for you and leave a comment if you need help!

    Thanks!

    [ayssocial_buttons id=”2″]