MeidokonWiki:

Shipyard - automated server builds as unique as you are

Anchor is full of smart people that hate boring work. Shipyard was created to relieve sysadmins of the mind-numbingly repetitive parts of server builds, leaving them free to do other things, like build more servers, or find more inventive ways to vent their rage through image macros.

This page makes use of the Seesaw plugin. Whenever you see a link preceded by » , that link will toggle the display of an annotation or other aid.

cat_halp.jpg

Contents

  1. Shipyard - automated server builds as unique as you are
    1. How shipyard fits into the workflow
    2. Assumed knowledge
    3. Things you absolutely must know about shipyard, even if you think it doesn't apply to you
    4. Workflow
    5. Using shipyard - Sales
    6. Using shipyard - Sysadmin
      1. How to use the checklist
      2. What about the robots? I was promised robots and blackjack
      3. Basic procedure
      4. Special build requirements

How shipyard fits into the workflow

Shipyard was designed to augment our existing L2 workflows as well as automate things. Once the sales team has hammered out a deal for the customer, they can input the tech specs directly into the system, then hand the build off for completion.

The end goal is to have end-to-end tracking and processing of server builds, from the time the customers e-signs the contract through to them receiving the handover email.

If you want more details, have a read of the related pages:

Assumed knowledge

Things you absolutely must know about shipyard, even if you think it doesn't apply to you

While shipyard has been created to be very robust, it interfaces with a lot of existing systems. This makes it susceptible to breaking in ways that aren't immediately obvious.

When a build is in progress (ie. still on the board and not yet marked Complete, it will interact with the admin system, provsys and RT, amongst other things. To avoid being set upon by rabid developers, we ask that you avoid certain actions:

Workflow

  1. Sales team creates a new order with the desired specs, notes for software to be installed, etc.

    • Extra servers may optionally be added to the same order, though in most cases one order == one server build.

  2. With the details now fixed, the order is passed to a sysadmin. This is handled by whoever is coordinating sysadmins, typically Pat.
  3. The sysadmin runs down the build sheet completing checklist items.
  4. The build will be passed to another sysadmin for QA before handover.
  5. Once the checklist is complete, the server build is all done and the customer has their server. Everyone can be happy.

Using shipyard - Sales

This section is not yet verified to be 100% accurate

Before you start:

Now you're ready to create an order. Perhaps you'd prefer a more graphical reference? A handy »annotated screenshot of the New Order page is also available.

shipyard_new_order_howto.png

  1. Head over to shipyard and follow the link to create a New Order

    • If you're asked to login, use your LDAP username and password
  2. Provide a descriptive name for the new task that will be created in the admin system
  3. Fill in the name of a parent task if applicable. This is a magic autocomplete field, so let it do its job. You may find it easiest to enter the parent task ID as a search term
  4. Note the account manager for the task if applicable
    • Prod a shipyard developer for assistance if you need a new account manager added

  5. Starting at Client Contact, fill in the relevant details

    • For existing customers, use the magic autocompleting form field to search for the desired contact. Searching for their email address is usually the easiest mechanism, but you can also use their name or contact ID

    • For new customers, go ahead and use the Create new contact button

  6. Unless the customer has specified other contacts, you should copy the contact details to the Billing and Tech contact fields using the provided buttons

  7. Fill in the required hardware specs for the server. The Product name dropdown will prefill most of the important fields for you, and should be used unless you've negotiated something special (rare)

  8. Provide the RT ticket number (eg. 123456) for the sales process. The sysadmin performing the build may need to refer to the details provided by the customer, so a reference to the ticket is vital.

  9. Ensure that the remaining bold-titled fields are filled in (Support level and Operating system)

  10. Tick the checkboxes for any "application" components that the customer has asked for
    • If there's no exact option available for what you need, please ask a shipyard developer for assistance. Incorrect details will cause confusion and errors when the server is being built

  11. Provide as many extra details as necessary in the Notes field

  12. Double-check your work, as it's not possible to change details of the order once it's submitted

  13. Push the Create order button and ask a coordinator to assign the build to a sysadmin

  14. Assign the sales ticket to the coordinator, and move it to the Provisioning queue

Does the customer's order require more than one server? Now is the time to add them to the order.

  1. Go to shipyard and select the order you just created, check this »screenshot if you're not sure what you're looking for

    shipyard_select_your_order.png

  2. Select the link for (new server build)

  3. Fill out the details as before, then Add build to order

Using shipyard - Sysadmin

You'll receive a nudge (probably from Pat) that a build has been assigned to you. If you're lucky, you'll even get a link to shipyard. Head to the shipyard and check your list of outstanding builds.

Find your server build and click its name to go to the checklist. It may be labelled "Unnamed" if a name hasn't yet been chosen. See »this screenshot if you're not sure what you're looking for.

shipyard_my_builds.png

You should now see a checklist that look somewhat like »this one . The sales person or coordinator may have already filled some fields out for you.

shipyard_new_checklist.png

How to use the checklist

  • The checklist can only be modified by whoever the build is assigned to, which can be changed using the controls in the top-right corner

    • The Take button will assign the build to you, allowing you to complete the checklist (duh)

    • The Automate button will assign the build to a robot, which is where the magic begins...

    • There are keyboard shortcuts for the buttons, just type T or A respectively

  • Checklist items are either a simple yes/no checkbox to indicate completion, or a text field with associated checkbox

  • You can't manually check a text field; type a value and hit <Enter> to "complete" the checklist item

  • Uncheck a text field by deleting the value and typing <Enter>

  • When a checklist step is completed, the name of the person who completed it is recorded on the far left of the row

What about the robots? I was promised robots and blackjack

Each checklist step is deliberately very straightforward, and some are simple enough to be automated. You'll notice in the above screenshot that some of the steps were done by "autobot", that's the codename of our robot.

There's a little icon to the left of each checklist step which denotes whether it can be automated. It's both an indicator as well as a control, clicking the icon toggles its state.

  • automation.gif Mudkip

    • Displayed: This is the default state for all steps, and indicates that automation will be attempted
    • Manually selected: Automate this step if possible

  • human.png L33t h4xx0r (that's you)

    • Displayed: This step cannot be automated, and requires a human to perform the work
    • Manually selected: Do not attempt to automate this step

When ownership of the build is passed to autobot it will start at the top of the checklist and work its way down, attempting to perform any incomplete steps. You'll see it update the checklist in realtime. If it encounters any errors it will stop, highlight the failed step, and disown the build. If a step can't be automated it will stop and disown the build, and mark that step as needing human action.

Incomplete steps are not coloured

Completed steps are highlighted in green

In-progress steps are highlighted in yellow

Failed steps are highlighted in red

If there's anything you don't want autobot to do, click the icon next to that step to toggle the automatability flag. It will stop when it gets there and proceed no further.

Basic procedure

  1. Take the build

  2. Read the specs and notes provided by the sales team, looking for anything that might need special attention. If there's any steps that shouldn't be automated, toggle them to "Human". See the next section for notes on special values

  3. Click the Automate button and watch until autobot hits something that it can't do

  4. Take ownership of the build again

  5. Either perform the step if it needs human interaction, or fix the problem so that it'll work
    • If you fixed a problem, check and then uncheck the step to clear its state, ready for another attempt at automation

    • If you're stuck, your next port of call is the mudpuppy log on wench 

      less /etc/service/mudpuppy/log/logs/current  # you can pipe this through tai64nlocal to get readable timestamps
      If you need help figuring out the problem, ask a shipyard developer or another sysadmin who's experienced in its voodoo

  6. Toggle the step away from "Human" if required, and click the Automate button again...

  7. Rinse and repeat until there's nothing you or autobot can do - typically when the server needs to be QA'ed. That's when you pass the build to a willing victim
  8. QA is easy:

    1. Take the build if your fellow sysadmin hasn't already palmed it off to you

    2. Enter your name in the box and press <Enter>

    3. Perform the QA process, linked from the checklist itself
    4. Get the other guy to fix any problems
    5. If it passes QA, check the box and return ownership of the build

  9. Once the build is complete (all steps are highlighted green), change the Build Status to Completed from the dropdown at the top of the page. You are done!

Special build requirements

Shipyard has so far been written to streamline the most common/useful cases of VPS builds. In doing so it takes away a lot of the input that we used to provide, and replaces it with intelligence gleaned from provsys and the admin system.

Some customers will have special requirements that override the automatically chosen values, which we'll cover here.

MeidokonWiki: furinkan/Anchor/writing/shipyard/howto (last edited 2011-11-06 21:26:00 by furinkan)