Shipyard - automated server builds as unique as you are

Anchor is full of smart people that hate boring work. Shipyard was created to relieve sysadmins of the mind-numbingly repetitive parts of server builds, leaving them free to do other things, like build more servers, or find more inventive ways to vent their rage through image macros.

How shipyard fits into the workflow

Shipyard was designed to augment our existing L2 workflows as well as automate things. Once the sales team has hammered out a deal for the customer, they can input the tech specs directly into the system, then hand the build off for completion.

The end goal is to have end-to-end tracking and processing of server builds, from the time the customers e-signs the contract through to them receiving the handover email.

If you want more details, have a read of the related pages:

• Technical overview

Assumed knowledge

• You know your way around the admin system, and understand basic concepts
  • Tasks, and parent-child relationships for tasks
  • Contacts
• Our various VPS product offerings, and ideally dedicated servers as well

Things you absolutely must know about shipyard, even if you think it doesn't apply to you

While shipyard has been created to be very robust, it interfaces with a lot of existing systems. This makes it susceptible to breaking in ways that aren't immediately obvious.

When a build is in progress (ie. still on the board and not yet marked Complete, it will interact with the admin system, provsys and RT, amongst other things. To avoid being set upon by rabid developers, we ask that you avoid certain actions:

• Don't merge related tasks in the admin system
• Don't rename related tasks in the admin system
• Don't delete related tasks in the admin system
• Don't rename any resources related to the build in the provisioning system

Workflow

1. Sales team creates a new order with the desired specs, notes for software to be installed, etc.

   • Extra servers may optionally be added to the same order, though in most cases one order == one server build.

2. With the details now fixed, the order is passed to a sysadmin. This is handled by whoever is coordinating sysadmins, typically Pat.
3. The sysadmin runs down the build sheet completing checklist items.
4. The build will be passed to another sysadmin for QA before handover.
5. Once the checklist is complete, the server build is all done and the customer has their server. Everyone can be happy.

Using shipyard

Sales

Before you start:

• You must have an RT sales ticket, it will be linked to the build
• You are about to create an order. An order has one or more server builds attached to it
• Shipyard will create a new task in the admin system for the order, you'll specify a parent task if the order comes from an existing customer

Now you're ready to create an order. Perhaps you'd prefer a more graphical reference? A handy »annotated screenshot of the New Order page »annotated screenshot of the New Order page is also available.

Show the procedure Show the procedure

Sysadmin

You'll receive a nudge (probably from Pat) that a build has been assigned to you. If you're lucky, you'll even get a link to shipyard. Head to the shipyard and check your list of outstanding builds.

Find your server build and click its name to go to the checklist. See this screenshot this screenshot if you're not sure what you're looking for.

You should now see a checklist that look somewhat like »this one »this one .

Shipyard Overview

• Many of these steps can be completed by an automated drone called Autobot Anything that cannot be completed by Autobot will have to be completed by a Sysadmin Work on a build can only be completed by the user assigned to the build. Autobot does work when builds are assigned to the 'autobot' user. Autobot will attempt to complete any items in the checklist that have all their dependencies completed. If Autobot comes to an item it Cannot Complete it will mark the item as Manual, or if it encounters and error while trying to process it, it will mark it as Failed.
• Checklist:
  • The checklist for a build is generated when the build is created.
  • Once it has been created, the options for the build cannot be changed.
  • Checklist Items can be in the following states:
    • White: Incomplete
    • Orange: In Progress
    • Red: Failed
    • Green: Complete
  • They also have an Automation state:
    • Mudpuppy: Attempt to Automate
      • Sysadmin: Do not attempt to Automate, must be completed manually
        • Checking a box that is Incomplete will set it to Complete
        • Checking a box that is Failed will set it to Incomplete

  • You can reach a build checklist directly via https://shipyard.engineroom.anchor.net.au/build/server/SERVERNAME

Build Instructions

• Go to https://shipyard.engineroom.anchor.net.au/

  • You will be presented with a list of builds that have been assigned to you (possibly empty) and a list of all the incomplete builds.
  • You can take unassigned builds by pressing the "Take" button next to it.
  • Once you have taken a build you can click on the server's name or "Unnamed" if it has not yet been assigned a name.
• Server Build view:

  • Take the build by pressing the Take button and set the Build Status to "In Progress" These can be found in the top right.

  • Please check the NOTES field at the top of the page. This lists any additional requirements or changes that need to be made during the build. It may require checklist items to be set to Manual mode if it will conflict with the automation.

  • Fill in any questions that require custom configuration. eg. custom server name or manually selecting a hypervisor, if you don't select anything, the autobot will pick for you.

  • If you wish for any steps to NOT be automated, you can click on the Mudpuppy(Mudkip) icon next to the step and it will turn into a SysadminNinja indicating that it should be completed by a sysadmin and not Autobot.

  • When you are ready for the Automation to proceed, Press the 'Automate' button in the top right.
  • The build will be assigned to 'autobot' and autobot will start taking items and attempting to complete them.
    • It it succeeds the item will turn green and autobot will continue on to the next item.

    • If it turns back to white with a SysAdmin Ninja icon it means the item cannot be completed by Mudpuppy and requires a SysAdmin to complete it.

      • This may be due to not having enough information available or the automation module being incomplete.
      • If it turns red there was an error and you should investigate the mudpuppy log on wench.
        • less /etc/service/mudpuppy/log/logs/current should give you some idea of what the problem was. It is possible the error will be in one of the subsystems mudpuppy calls, such as Orchestra or the AdminDB SimpleAPI, you may need to check their logs.
          • You can assign the build back to yourself with the Take button and press the checkbox next to the item twice to clear the failed state once you have fixed the problem. Then GOTO 10. Or manually complete the item.
      • Once manually completing an item, assign it back to autobot to do as much of the build as it can.
    • When you reach the QA step (Note: Not all builds will include this step), get someone else to assign to QA, fill their name in the box and have them complete the QA section.
    • When the build is complete, change Build Status to 'Completed'

QA instructions

• As a QA drone it is your job to ensure that the machine is in a usable state and that the build matches the requirements and all steps were completed.
• Check the draft build email contains the correct information.
• Tick the Final QA checkbox
• Assign the build back to the builder and notify them.