Shipyard - automated server builds as unique as you are

Anchor is full of smart people that hate boring work. Shipyard was created to relieve sysadmins of the mind-numbingly repetitive parts of server builds, leaving them free to do other things, like build more servers, or find more inventive ways to vent their rage through image macros.

How shipyard fits into the workflow

Shipyard was designed to augment our existing L2 workflows as well as automate things. Once the sales team has hammered out a deal for the customer, they can input the tech specs directly into the system, then hand the build off for completion.

The end goal is to have end-to-end tracking and processing of server builds, from the time the customers e-signs the contract through to them receiving the handover email.

If you want more details, have a read of the related pages:

• Technical overview

Assumed knowledge

• You know your way around the admin system, and understand basic concepts
  • Tasks, and parent-child relationships for tasks
  • Contacts
• Our various VPS product offerings, and ideally dedicated servers as well

Workflow

1. Sales team creates a new order with the desired specs, notes for software to be installed, etc.

   • Extra servers may optionally be added to the same order, though in most cases one order == one server build.

2. With the details now fixed, the order is passed to a sysadmin. This is handled by whoever is coordinating sysadmins, typically Pat.
3. The sysadmin runs down the build sheet completing checklist items.
4. Once the checklist is complete, the server build is all done and everyone can be happy.

Using shipyard

Sales

1. Head over to shipyard and follow the link to create a New Order

2. Provide a descriptive name for the new task that will be created in the admin system
3. Fill in the parent task if applicable. This is a magic autocomplete field, so let it do its job
4. Note the account manager for the task, prod a shipyard developer for assistance if you need a new account manager added

5. Starting at Client Contact, fill in the relevant details

   • For existing customers, use the magic autocompleting form field to search for the desired contact. Searching for their email address is usually the easiest mechanism

   • For new customers, go ahead and use the Create new contact button

6. Unless the customer has specified other contacts, you should copy the contact details to the Billing and Tech contact fields using the provided buttons

7. Fill in the required hardware specs for the server. The Product name dropdown will prefill most of the important fields for you, and should be used unless you've negotiated something special (rare)

8. Provide the RT ticket number (eg. 123456) for the sales process. The sysadmin performing the build may need to refer to the details provided by the customer, so a reference to the ticket is vital.

9. Ensure that the remaining bold-titled fields are filled in (Support level and Operating system)

10. Tick the checkboxes for any "application" components that the customer has asked for

11. Provide as many extra details as necessary in the Notes field

12. Double-check your work, as it's not possible to change details of the order once it's submitted

13. Push the Create order button and ask a coordinator to assign the build to a sysadmin

Shipyard Overview

• Shipyard is a Web based User Interface to Make-magic and Mudpuppy (commonly known as Autobot) In Shipyard you can create new build orders with custom requirements and it will generate a checklist of steps to complete. Many of these steps can be completed by an automated drone called Autobot Anything that cannot be completed by Autobot will have to be completed by a Sysadmin Work on a build can only be completed by the user assigned to the build. Autobot does work when builds are assigned to the 'autobot' user. Autobot will attempt to complete any items in the checklist that have all their dependencies completed. If Autobot comes to an item it Cannot Complete it will mark the item as Manual, or if it encounters and error while trying to process it, it will mark it as Failed.
• Checklist:
  • The checklist for a build is generated when the build is created.
  • Once it has been created, the options for the build cannot be changed.
  • Checklist Items can be in the following states:
    • White: Incomplete
    • Orange: In Progress
    • Red: Failed
    • Green: Complete
  • They also have an Automation state:
    • Mudpuppy: Attempt to Automate
      • Sysadmin: Do not attempt to Automate, must be completed manually
        • Checking a box that is Incomplete will set it to Complete
        • Checking a box that is Failed will set it to Incomplete

  • You can reach a build checklist directly via https://shipyard.engineroom.anchor.net.au/build/server/SERVERNAME

New Order Instructions

For each order there is by default one build. You can add more later if needed (You should keep linked builds together on the same order rather than creating new ones).

• Before creating a new order in Shipyard an RT ticket must exist.

• Go to https://shipyard.engineroom.anchor.net.au/ and log in with your LDAP details

• Press the "New Order" Button

• Give the Task a descriptive name. A new admin task must be created for each new order.

• Set the parent task if appropriate ( you can search via name or by task ID )
• Add the contacts for the new task, need Client, Billing and Tech (can all the be same if necessary) (can search by name, email or Contact ID)
• You can create new contacts also if you need to by pressing the "Create new contact" button.
• Create the first server build for this order
  • (Optional) Select a product from the list which will fill in default values for the below items.
  • Set the priority and deadline date. (Please set the deadline so the builder knows when it must be completed by)

  • Set the build requirements Do not select incorrect options and leave notes to the contrary in the Notes field. If a required option is not available, please notify one of the development team in #dev or in person. Do not proceed.

  • Select any services required to be installed.
  • Add any ADDITIONAL items to notes, the notes must not contradict what is in the selected options since they are used by automation systems and cannot be changed after creating the build order.
• Press the Create Order button.

* If required, you may add additional server builds to the order.

• Adding new server build to an order.

• Go to https://shipyard.engineroom.anchor.net.au/

• Click on the Order number you wish to append to. (Has an icon of a page with a graph)
• Press the '(new server build)' link
• Fill in the details as above.

Build Instructions

• Go to https://shipyard.engineroom.anchor.net.au/

  • You will be presented with a list of builds that have been assigned to you (possibly empty) and a list of all the incomplete builds.
  • You can take unassigned builds by pressing the "Take" button next to it.
  • Once you have taken a build you can click on the server's name or "Unnamed" if it has not yet been assigned a name.
• Server Build view:

  • Take the build by pressing the Take button and set the Build Status to "In Progress" These can be found in the top right.

  • Please check the NOTES field at the top of the page. This lists any additional requirements or changes that need to be made during the build. It may require checklist items to be set to Manual mode if it will conflict with the automation.

  • Fill in any questions that require custom configuration. eg. custom server name or manually selecting a hypervisor, if you don't select anything, the autobot will pick for you.

  • If you wish for any steps to NOT be automated, you can click on the Mudpuppy(Mudkip) icon next to the step and it will turn into a SysadminNinja indicating that it should be completed by a sysadmin and not Autobot.

  • When you are ready for the Automation to proceed, Press the 'Automate' button in the top right.
  • The build will be assigned to 'autobot' and autobot will start taking items and attempting to complete them.
    • It it succeeds the item will turn green and autobot will continue on to the next item.

    • If it turns back to white with a SysAdmin Ninja icon it means the item cannot be completed by Mudpuppy and requires a SysAdmin to complete it.

      • This may be due to not having enough information available or the automation module being incomplete.
      • If it turns red there was an error and you should investigate the mudpuppy log on wench.
        • less /etc/service/mudpuppy/log/logs/current should give you some idea of what the problem was. It is possible the error will be in one of the subsystems mudpuppy calls, such as Orchestra or the AdminDB SimpleAPI, you may need to check their logs.
          • You can assign the build back to yourself with the Take button and press the checkbox next to the item twice to clear the failed state once you have fixed the problem. Then GOTO 10. Or manually complete the item.
      • Once manually completing an item, assign it back to autobot to do as much of the build as it can.
    • When you reach the QA step (Note: Not all builds will include this step), get someone else to assign to QA, fill their name in the box and have them complete the QA section.
    • When the build is complete, change Build Status to 'Completed'

QA instructions

• As a QA drone it is your job to ensure that the machine is in a usable state and that the build matches the requirements and all steps were completed.
• Check the draft build email contains the correct information.
• Tick the Final QA checkbox
• Assign the build back to the builder and notify them.

Important Operational Notes

• You can not touch any details of a build in progress in any system (provsys, adminsys, etc). The build is in progress until it is set complete! This means:
  1. No merging related tasks.
  2. No renaming related tasks.
  3. No deleting related tasks.
  4. No renaming anything related in provsys.