How to install Sonic on Celestica dx010 100g switch

In this document i am going to talk about installation of Sonic Network operating system on celestica dx010 32x100G switches 
 If you do these steps you'll end up with a l2 switch that can forward 100g per port. Keep in mind that Spanning tree is still an issue on Sonic. 
 Important : Make sure you check the Sonic HCL, if your model is not there, don't try to do this installation. 
 Upgrading or re-installing onie 
 Onie is the Network operating system loader created by open network computing. It is a software to create zero touch provisioning on open compute / Bare metal switches. there are some binaries supplied by open compute platform, and you can compile or cross compile your version of it. 
 By your version i meant the model of the bare metal switch. before 2016 ONL which is sonic based on, supported powerpc cpu models however with sonic you are stuck with intel and some arm cpu's, no powerpc cpu is supported or can be compiled. 
 Compiling onie 
 To compile onie you need to read this documentation and have many headaches. I can't get in to details here because it will use a lot of space but it is a good documentation and pretty straight-forward to do. 
 Onie build documentation 
 Basically it will create a folder with many permutations of onie for every device out there. 
 if your device is a Powerpc platform then you'll need to do a cross-compile 
 installation or upgrading onie using u-boot 
 After powercycle press any key to stop u-boot process. take backup of current environment using 
 printenv
 
 output should look like below, please copy and paste it somwhere safe. 
 
 Click for environment details 
 ```=> printenv
active=image2
autoload=no
backup=image1
baudrate=115200
bootactive=fsload 1000000 $active; bootm 1000074
bootargs=root=dev/ram console=ttyS0,115200
bootbackup=fsload 1000000 $backup; bootm 1000074
bootcmd=run check_boot_reason; run nos_bootcmd; run onie_bootcmd
bootdelay=3
bootfile=uImage
bootimage1=fsload 1000000 image1; bootm 1000074
bootimage2=fsload 1000000 image2; bootm 1000074
check_boot_reason=if test -n $onie_boot_reason; then setenv onie_bootargs boot_reason=$onie_boot_rea
son; run onie_bootcmd; fi;
consoledev=ttyS0
dboot=setenv bootargs root=/dev/ram mtdparts=physmap-flash.0:126336k(jffs2),4096k(onie),128k(u-boot-
env),512k(u-boot) quiet console=ttyS0,$baudrate $dbootfrom && bootimg $dkaddr $draddr $ddaddr
ddaddr=0x0D000000
dhcp_user-class=powerpc-quanta_ly2r-r0_uboot
dhcp_vendor-class-identifier=powerpc-quanta_ly2r-r0
diag_bootcmd=run dsdboot || run flash_bootcmd || run dnetboot
dkaddr=0x0A000000
dmemerase=mw $dkaddr 0 100; mw $draddr 0 100; mw $ddaddr 0 100; setenv dbootfrom
dnetboot=run dmemerase; run dnetcheck; if test -n $dbootfrom && test $dbootfrom = netboot; then run 
dboot; fi
dnetcheck=echo Checking diag from tftp server...; if tftp $dkaddr uImage && tftp $draddr uInitrd2m &
& tftp $ddaddr ly2r.dtb; then setenv dbootfrom netboot; fi
draddr=0x0B000000
dsdboot=run dmemerase; run dsdcheck; if test -n $dbootfrom && test $dbootfrom = sdboot; then run dbo
ot; fi
dsdcheck=echo Checking diag from Secure Digital Card...; if mmcinfo && ext2load mmc 0:1 $dkaddr uIma
ge && ext2load mmc 0:1 $draddr uInitrd2m && ext2load mmc 0:1 $ddaddr ly2r.dtb; then setenv dbootfrom
 sdboot; fi
ethact=eTSEC1
ethaddr=a8:1e:84:58:67:10
flash_bootcmd=setenv bootargs root=/dev/ram mtdparts=physmap-flash.0:58368k(jffs2),3968k(ramdisk),30
72k(kernel),128k(dtb),60800k(empty),4096k(onie),128k(u-boot-env),512k(u-boot) console=ttyS0,$baudrat
e; bootm EBCE0000 EB900000 EBFE0000
hostname=LY2RP2020
hwconfig=usb1:dr_mode=host,phy_type=ulpi,esdhc
ipaddr=192.168.2.1
loadaddr=1000000
loads_echo=1
mtddevnum=0
mtdids=nor0=nor0 [0/0]
mtdparts=nor0:65536k(jffs2),60800k(open),4096k(ONIE),128k(u-boot-env),512k(u-boot)
nfsboot=setenv bootargs root=/dev/nfs rw nfsroot=$serverip:$rootpath ip=$ipaddr:$serverip:$gatewayip:$netmask:$hostname:$netdev:off console=$consoledev,$baudrate $othbootargs;tftp $loadaddr $bootfile;tftp $fdtaddr $fdtfile;bootm $loadaddr - $fdtaddr
nos_bootcmd=run bootactive; run bootbackup; run bootimage1; run bootimage2
onie_args=run onie_initargs onie_platformargs
onie_bootcmd=echo Loading Open Network Install Environment ...; echo Platform: $onie_platform ; echo Version : $onie_version ; cp.b $onie_start $loadaddr ${onie_sz.b} && run onie_args && bootm ${loadaddr}
onie_dropbear_dss_host_key=begin-base64@600@d#AAAAB3NzaC1kc3MAAACBAOCuKIyK2CzC6aUUNVfJLj6SlJ+anzWWUbuqtyn4#oty/BYzbG7rk+9exB/I9SOBMf0n8BHGJOdS+hXVFe8CAGf0fI7AlHF0zTZCN#kZAW70T175b4AZC8zCegABBKv86s0s7OT4qOXLCeYjCM9gMWHRDAYHDfzktY#vBHapERPhIHzAAAAFQDfAxkC7+W/gxwO8BAg9Se08laRBwAAAIEAyjX1GFoX#oqPyEVn/qQ85xXpo3fr3UNT/NmZZGXQFZ+pUnHDOO6OwTZB0XLpPxzQ8WUVJ#Bt9d2tLC5Sw6rCp7odbp0Od9uWOc7ljt7HKtQVuodB/yjt9KbQc69sOY25bC#efY2c0IcZ9LiNbQbH4COR6zGMLzA2XgHQ7uJ7xmV1fsAAACBAJP+ZxxFAehm#H9Ac5HUHYG4ithFYK7l5MHRrA/pEGuqwUeezUKO9B8soCP9r4SoT/vLzUxQ5#TMwyANyi7BRjKnkjrRjJ/r6o9t5PS7YuGm/3G6hOL7UHJoDIDFuznL9NG+f4#GhHF6NElGGnpmATry0+5gTN2KDYlZLoMriGaGBLVAAAAFQCDpjU05iU1Lbci#IvGrJfEo2HQweg==#====#
onie_dropbear_rsa_host_key=begin-base64@600@r#AAAAB3NzaC1yc2EAAAADAQABAAAAgQC8AVEAfeuMSVYNPJPULsBYL0KrFOOM#62CmnWNODmX230TZYW/DDv/NSOMUNFUR109MpzEyy8WmNqqNl+h0X3dK/KUy#H4iZhIxXXYaRu8Yywfaf/xXNnCcQqdj5JYmNGlaRggcvkbYhnX2Hus7LrkGP#k0xdkl5erOsUGPmnf5RLNQAAAIAqqn3Qh8MiBi6iOGIwICAEo00U+FufyncX#uNYli7z2JrtAPg2YDdpdozHtjubTcrKLadIPfhEv5G4WTigx6Mxwcw1oHUon#LoBzMCEjULMpyReO5xQcyV0WTLyPncPgqh4wtT7gr2ePkEASwMgcIiXu0tV9#Q81k8UOcMZQz5kc3QQAAAEEA8mO+xK2LBWrmT37yh3c4xbg/Naj60CPxzRKR#HF0w6FzL8BcCAL009z/YHtIJVpoXo321c8DNazZ47hQEoQpv0QAAAEEAxo/Q#1zXuEwsJUqrOE5RZ8KdfsQSEw9I+Pl8pDNBV06MTbnQPgjMr9beD6NfSo1po#uZmbPVzaakzOtyduVCqCJQ==#====#
onie_initargs=setenv bootargs quiet console=$consoledev,$baudrate
onie_machine=quanta_ly2r
onie_machine_rev=0
onie_platform=powerpc-quanta_ly2r-r0
onie_platformargs=setenv bootargs $bootargs serial_num=${serial#} eth_addr=$ethaddr $onie_bootargs $onie_debugargs 
onie_rescue=setenv onie_boot_reason rescue && boot 
onie_start=0xefb60000
onie_sz.b=0x00400000
onie_uninstall=setenv onie_boot_reason uninstall && boot
onie_update=setenv onie_boot_reason update && boot 
onie_vendor_id=7244
onie_version=2015.05-v1.2-9d597eb
partition=nor0,0
platform=quanta_ly2r
ramboot=setenv bootargs root=/dev/ram rw console=$consoledev,$baudrate $othbootargs; tftp $ramdiskaddr $ramdiskfile;tftp $loadaddr $bootfile;tftp $fdtaddr $fdtfile;bootm $loadaddr $ramdiskaddr $fdtaddr
rootpath=/opt/nfsroot
serial#=QTFCEA7070054
serverip=192.168.2.11
stderr=serial
stdin=serial
stdout=serial
ver=U-Boot 2010.12 (Oct 14 2016 - 16:28:02) - ONIE 2015.05-v1.2-9d597eb

Environment size: 5059/8188 bytes```
 
 
 We are going to change environment as shown and explanied below, 
 Set ip address to eth0/management port of bare metal switch 
 setenv ipaddr 4.3.2.2 
setenv netmask 255.255.255.252
 
 Specify tftp server address 
 setenv serverip 4.3.2.1
 
 save changes 
 save
 
 Make sure have the same subnet with tftp server
After this point all the procedure will be destructive, please keep in mind that you might loose the current boot configuration or configurations within device. 
 specify space in flash 
 setenv start 0xefb60000
setenv sz.b 0x4a0000
 
 upload file to ram 
 tftp onie-quanta_ly2r-r0.bin
 
 Disable protection on flash and erase the old onie 
 protect off $start +${sz.b} && erase $start +${sz.b}
 
 start flashing 
 cp.b $fileaddr $start ${sz.b} && protect on $start +${sz.b}
 
 after this point Onie will be upgraded/installed and procedure will reset the device booting into new onie and it's environment 
 Installing sonic 
 Before install 
 Sonic is a buggy software, there are many missing features, and because of that there are many builds for every vendor
the list is here 
 we are going to use 202111 edition on celestica dx010 
 installation 
 Onie will boot up and you will be greeted with onie-discovery process, unless you did special dhcp configurations, we will need to stop onie-discovery 
 onie-discover-stop
 
 then the ip addressing 
 ip addr add 4.3.2.2/30 dev eth0
ip link set up dev eth0
 
 start installing sonic 
 onie-nos-install tftp://4.3.2.1/sonic-buildimage.broadcom
 
 at this point onie installer will do its thing after 5-10 minutes you will be greeted with sonics login screen 
 After install 
 Default user name: admin
default pass : YourPaSsWoRd
By default sonic names every 100g interface with steps of 4,
Ethernet0,Ethernet4,Ethernet8 and there are aliases for you to see which port are you working on 
 admin@sonic:~$ show interfaces status
Interface Lanes Speed MTU FEC Alias Vlan Oper Admin Type Asym PFC
 ----------- --------------- ------- ----- ----- ------- ------ ------ ------- -------------- ----------
Ethernet0 65 10G 9100 N/A Eth1/1 routed down down QSFP+ or later N/A
Ethernet1 66 10G 9100 N/A Eth1/2 routed down down QSFP+ or later N/A
Ethernet2 67 10G 9100 N/A Eth1/3 routed down down QSFP+ or later N/A
Ethernet3 68 10G 9100 N/A Eth1/4 routed down down QSFP+ or later N/A
Ethernet4 69 10G 9100 N/A Eth2/1 routed down down QSFP+ or later N/A
Ethernet5 70 10G 9100 N/A Eth2/2 routed down down QSFP+ or later N/A
Ethernet6 71 10G 9100 N/A Eth2/3 routed down down QSFP+ or later N/A
Ethernet7 72 10G 9100 N/A Eth2/4 routed down down QSFP+ or later N/A
Ethernet8 73,74,75,76 100G 9100 N/A Eth3 routed down up N/A N/A
 Ethernet12 77,78,79,80 100G 9100 N/A Eth4 routed down up N/A N/A
 Ethernet16 33,34,35,36 100G 9100 N/A Eth5 routed down up N/A N/A
 Ethernet20 37,38,39,40 100G 9100 N/A Eth6 routed down up N/A N/A
 Ethernet24 41,42,43,44 100G 9100 N/A Eth7 routed down up N/A N/A
 Ethernet28 45,46,47,48 100G 9100 N/A Eth8 routed down up N/A N/A
 Ethernet32 49,50,51,52 100G 9100 N/A Eth9 routed down up N/A N/A
 Ethernet36 53,54,55,56 100G 9100 N/A Eth10 routed down up N/A N/A
 Ethernet40 57,58,59,60 100G 9100 N/A Eth11 routed down up N/A N/A
 Ethernet44 61,62,63,64 100G 9100 N/A Eth12 routed down up N/A N/A
 Ethernet48 81,82,83,84 100G 9100 N/A Eth13 routed down up N/A N/A
 Ethernet52 85,86,87,88 100G 9100 N/A Eth14 routed down up N/A N/A
 Ethernet56 89,90,91,92 100G 9100 N/A Eth15 routed down up N/A N/A
 Ethernet60 93,94,95,96 100G 9100 N/A Eth16 routed down up N/A N/A
 Ethernet64 97,98,99,100 100G 9100 N/A Eth17 routed down up N/A N/A
 Ethernet68 101,102,103,104 100G 9100 N/A Eth18 routed down up N/A N/A
 Ethernet72 105,106,107,108 100G 9100 N/A Eth19 routed down up N/A N/A
 Ethernet76 109,110,111,112 100G 9100 N/A Eth20 routed down up N/A N/A
 Ethernet80 1,2,3,4 100G 9100 N/A Eth21 routed down up N/A N/A
 Ethernet84 5,6,7,8 100G 9100 N/A Eth22 routed down up N/A N/A
 Ethernet88 9,10,11,12 100G 9100 N/A Eth23 routed down up N/A N/A
 Ethernet92 13,14,15,16 100G 9100 N/A Eth24 routed down up N/A N/A
 Ethernet96 17 25G 9100 N/A Eth25/1 routed down down QSFP+ or later N/A
 Ethernet97 18 25G 9100 N/A Eth25/2 routed down down QSFP+ or later N/A
 Ethernet98 19 25G 9100 N/A Eth25/3 routed down down QSFP+ or later N/A
 Ethernet99 20 25G 9100 N/A Eth25/4 routed down down QSFP+ or later N/A
Ethernet100 21,22,23,24 100G 9100 N/A Eth26 routed down up N/A N/A
Ethernet104 25,26,27,28 100G 9100 N/A Eth27 routed down up N/A N/A
Ethernet108 29,30,31,32 40G 9100 N/A Eth28 routed down up QSFP+ or later N/A
Ethernet112 113,114,115,116 100G 9100 N/A Eth29 routed down up N/A N/A
Ethernet116 117,118,119,120 40G 9100 N/A Eth30 routed down up QSFP+ or later N/A
Ethernet120 121,122,123,124 100G 9100 N/A Eth31 routed down up N/A N/A
Ethernet124 125,126,127,128 100G 9100 N/A Eth32 routed down up N/A N/A
 
 Sonic has some problems, it's not the news, it's a fact. One of them is this installation media doesn't create a running environment, to do that follow the steps below 
 sudo sonic-cfggen --preset l2 -p -H -k Seastone-DX010 -w
sudo config reload -y
sudo reboot
 
 
 Clear the IP addresses from each interface 
 show runningconfiguration interfaces | grep \| | awk -F'"' '{ print $2 }' | awk -F'|' '{ print "sudo config interface ip remove "$1" "$2 }' > /var/tmp/remove-l3-ips
bash /var/tmp/remove-l3-ips
rm -f /var/tmp/remove-l3-ips
 
 
 Configure sonic 
 Layer1 
 Every port and platform or asic has different abilities or features, for example you can breakout every port on a celestica dx010 how ever an edge-core AS9716-32D will only do breakouts on Ethernet0-20 which is the top row of switch.
I was able to work with 3rdparty qsfp modules on sonic however i couldn't find an HCL about qsfps . 
 Breakouts 
 You neeed to do everything written on after install to be able to do breakouts.
Supported breakouts are ; 
 
 1x100G[100g] 
 1x100G[40g] 
 2x50G[50G] 
 4x25G[25G] 
 4x25G[10G] 
 
 to break out 1 port to 4 10g ports 
 sudo config interface breakout Ethernet96 4x25G[10G] -y
 
 a normal output should look like below, after this you will need to reboot the switch. 
 admin@sonic:~$ sudo config interface breakout Ethernet96 4x25G[10G] -y

Running Breakout Mode : 1x100G[40G] 
Target Breakout Mode : 4x25G[10G]

Ports to be deleted : 
 {
 "Ethernet96": "100000"
}
Ports to be added : 
 {
 "Ethernet96": "25000",
 "Ethernet97": "25000",
 "Ethernet98": "25000",
 "Ethernet99": "25000"
}

After running Logic to limit the impact

Final list of ports to be deleted : 
 {
 "Ethernet96": "100000"
} 
Final list of ports to be added : 
 {
 "Ethernet96": "25000",
 "Ethernet97": "25000",
 "Ethernet98": "25000",
 "Ethernet99": "25000"
}
sonic_yang(6):Note: Below table(s) have no YANG models: SNMP, SNMP_COMMUNITY
Below Config can not be verified, It may cause harm to the system
 {}
Do you wish to Continue? [y/N]: y
Breakout process got successfully completed.
Please note loaded setting will be lost after system reboot. To preserve setting, run `config save`.
 
 Dont forget to save :) 
 admin@sonic:~$ sudo config save
Existing files will be overwritten, continue? [y/N]: y
Running command: /usr/local/bin/sonic-cfggen -d --print-data > /etc/sonic/config_db.json
 
 You will reboot the switch so it can show newly emerged ports. 
 After reboot you will have Ethernet 96 to 99, set the speeed to 10g on those interfaces 
 config interface speed Ethernet96 10000
 
 And start the port, 
 config interface startup Ethernet96
 
 Do this for every port you broke and want to work immediately 
 Layer2 
 Creating Vlans 
 here 
 sudo config vlan add 1000 	 
 
 vlans and membership 
 to see the details 
 show vlan brief
 
 tagging, untagged ports 
 
 
 tagged 
 sudo config vlan member add 1000 Ethernet48
 
 
 
 untagged 
 sudo config vlan member add -u 1000 Ethernet48
 
 
 
 You can use the following commands to edit ports in bulk 
 seq 36 39|xargs -i -t sudo config interface speed Ethernet{} 10000 
seq 36 39|xargs -i -t sudo config interface startup Ethernet{} 
seq 36 39|xargs -i -t sudo config vlan member add -u 1000 Ethernet{} 
 
 Layer3 
 intervlan routing 
 to add an ip or svi to vlan 1 use the command below 
 admin@sonic:~$ sudo config interface ip add Vlan1000 192.168.1.2/24 
 
 routed interfaces 
 as in svi configuration but this time use the port name 
 sudo config interface ip add Ethernet4 192.168.1.2/24