= Shipyard - automated server builds as unique as you are = Anchor is full of smart people that hate boring work. Shipyard was created to relieve sysadmins of the mind-numbingly repetitive parts of server builds, leaving them free to do other things, like build more servers, or find more inventive ways to vent their rage through image macros. {{{#!wiki note This page makes use of the [[http://moinmo.in/MacroMarket/SeeSaw| Seesaw]] plugin. Whenever you see a link preceded by ~+'''ยป'''+~, that link will toggle the display of an annotation or other aid. }}} <> == How shipyard fits into the workflow == Shipyard was designed to '''augment''' our existing L2 workflows as well as '''automate''' things. Once the sales team has hammered out a deal for the customer, they can input the tech specs directly into the system, then hand the build off for completion. The end goal is to have end-to-end tracking and processing of server builds, from the time the customers e-signs the contract through to them receiving the handover email. If you want more details, have a read of the related pages: * [[/../Technical_overview| Technical overview]] == Assumed knowledge == * You know your way around the admin system, and understand basic concepts * Tasks, and parent-child relationships for tasks * Contacts * Our various VPS product offerings, and ideally dedicated servers as well == Things you absolutely must know about shipyard, even if you think it doesn't apply to you == While shipyard has been created to be very robust, it interfaces with a lot of existing systems. This makes it susceptible to breaking in ways that aren't immediately obvious. When a build is in progress (ie. still on the board and not yet marked ''Complete'', it will interact with the admin system, provsys and RT, amongst other things. To avoid being set upon by rabid developers, we ask that you avoid certain actions: * Don't merge related tasks in the admin system * Don't rename related tasks in the admin system * Don't delete related tasks in the admin system * Don't rename any resources related to the build in the provisioning system == Workflow == 1. Sales team [[https://shipyard.engineroom.anchor.net.au/build/order/new/| creates a new order]] with the desired specs, notes for software to be installed, etc. * Extra servers may optionally be added to the same order, though in most cases ''one order == one server build''. 1. With the details now fixed, the order is passed to a sysadmin. This is handled by whoever is coordinating sysadmins, typically Pat. 1. The sysadmin runs down the build sheet completing checklist items. 1. The build will be passed to another sysadmin for QA before handover. 1. Once the checklist is complete, the server build is all done and the customer has their server. Everyone can be happy. == Using shipyard == === Sales === {{{#!wiki note This section is not yet verified to be 100% accurate }}} Before you start: * You must have an RT sales ticket, it will be linked to the build * You are about to create an order. An order has one or more server builds attached to it * Shipyard will create a new task in the admin system for the order, you'll specify a parent task if the order comes from an existing customer Now you're ready to create an order. Perhaps you'd prefer a more graphical reference? A handy <> is also available. {{{#!wiki seesaw shipyard_new_order_howto {{attachment:shipyard_new_order_howto.png}} }}} <> {{{{#!wiki seesaw sales_process 1. Head over to [[https://shipyard.engineroom.anchor.net.au/| shipyard]] and follow the link to create a [[https://shipyard.engineroom.anchor.net.au/build/order/new/| New Order]] * If you're asked to login, use your LDAP username and password 1. Provide a descriptive name for the new task that will be created in the admin system 1. Fill in the name of a parent task if applicable. This is a magic autocomplete field, so let it do its job. You may find it easiest to enter the parent task ID as a search term 1. Note the account manager for the task if applicable * Prod a shipyard developer for assistance if you need a new account manager added 1. Starting at ''Client Contact'', fill in the relevant details * For existing customers, use the magic autocompleting form field to search for the desired contact. Searching for their '''email address''' is usually the easiest mechanism, but you can also use their name or contact ID * For new customers, go ahead and use the ''Create new contact'' button 1. Unless the customer has specified other contacts, you should copy the contact details to the ''Billing'' and ''Tech'' contact fields using the provided buttons 1. Fill in the required hardware specs for the server. The '''Product name''' dropdown will prefill most of the important fields for you, and should be used unless you've negotiated something special (rare) 1. Provide the RT ticket number (eg. ''123456'') for the sales process. The sysadmin performing the build may need to refer to the details provided by the customer, so a reference to the ticket is vital. 1. Ensure that the remaining bold-titled fields are filled in (''Support level'' and ''Operating system'') 1. Tick the checkboxes for any "application" components that the customer has asked for * If there's no exact option available for what you need, please ask a shipyard developer for assistance. Incorrect details will cause confusion and errors when the server is being built 1. Provide as many extra details as necessary in the ''Notes'' field 1. '''Double-check your work, as it's not possible to change details of the order once it's submitted''' 1. Push the ''Create order'' button and ask a coordinator to assign the build to a sysadmin 1. Assign the sales ticket to the coordinator, and move it to the ''Provisioning'' queue Does the customer's order require more than one server? Now is the time to add them to the order. 1. Go to [[https://shipyard.engineroom.anchor.net.au/| shipyard]] and select the order you just created, check this <> if you're not sure what you're looking for {{{#!wiki seesaw shipyard_select_your_order {{attachment:shipyard_select_your_order.png}} }}} 1. Select the link for ''(new server build)'' 1. Fill out the details as before, then ''Add build to order'' }}}} === Sysadmin === You'll receive a nudge (probably from Pat) that a build has been assigned to you. If you're lucky, you'll even get a link to shipyard. Head to [[https://shipyard.engineroom.anchor.net.au/build/| the shipyard]] and check your list of outstanding builds. Find your server build and click its name to go to the checklist. See <> if you're not sure what you're looking for. {{{#!wiki seesaw shipyard_my_builds {{attachment:shipyard_my_builds.png}} }}} You should now see a checklist that look somewhat like <>. {{{#!wiki seesaw shipyard_new_checklist {{attachment:shipyard_new_checklist.png}} }}} === Shipyard Overview === Many of these steps can be completed by an automated drone called Autobot Anything that cannot be completed by Autobot will have to be completed by a Sysadmin Work on a build can only be completed by the user assigned to the build. Autobot does work when builds are assigned to the 'autobot' user. Autobot will attempt to complete any items in the checklist that have all their dependencies completed. If Autobot comes to an item it Cannot Complete it will mark the item as Manual, or if it encounters and error while trying to process it, it will mark it as Failed. * Checklist: * The checklist for a build is generated when the build is created. * Once it has been created, the options for the build cannot be changed. * Checklist Items can be in the following states: * White: Incomplete * Orange: In Progress * Red: Failed * Green: Complete * They also have an Automation state: * Mudpuppy: Attempt to Automate * Sysadmin: Do not attempt to Automate, must be completed manually * Checking a box that is Incomplete will set it to Complete * Checking a box that is Failed will set it to Incomplete * You can reach a build checklist directly via https://shipyard.engineroom.anchor.net.au/build/server/SERVERNAME === Build Instructions === * Go to https://shipyard.engineroom.anchor.net.au/ * You will be presented with a list of builds that have been assigned to you (possibly empty) and a list of all the incomplete builds. * You can take unassigned builds by pressing the "Take" button next to it. * Once you have taken a build you can click on the server's name or "Unnamed" if it has not yet been assigned a name. * Server Build view: * Take the build by pressing the ''Take'' button and set the ''Build Status'' to "In Progress" These can be found in the top right. * Please check the ''NOTES'' field at the top of the page. This lists any additional requirements or changes that need to be made during the build. It may require checklist items to be set to ''Manual'' mode if it will conflict with the automation. * Fill in any questions that require custom configuration. eg. custom server name or manually selecting a hypervisor, if you don't select anything, the autobot will pick for you. * If you wish for any steps to NOT be automated, you can click on the Mudpuppy(Mudkip) icon next to the step and it will turn into a SysadminNinja indicating that it should be completed by a sysadmin and not Autobot. * When you are ready for the Automation to proceed, Press the 'Automate' button in the top right. * The build will be assigned to 'autobot' and autobot will start taking items and attempting to complete them. * It it succeeds the item will turn green and autobot will continue on to the next item. * If it turns back to white with a [[SysAdmin Ninja]] icon it means the item cannot be completed by Mudpuppy and requires a SysAdmin to complete it. * This may be due to not having enough information available or the automation module being incomplete. * If it turns red there was an error and you should investigate the mudpuppy log on wench. * less /etc/service/mudpuppy/log/logs/current should give you some idea of what the problem was. It is possible the error will be in one of the subsystems mudpuppy calls, such as Orchestra or the AdminDB SimpleAPI, you may need to check their logs. * You can assign the build back to yourself with the Take button and press the checkbox next to the item twice to clear the failed state once you have fixed the problem. Then GOTO 10. Or manually complete the item. * Once manually completing an item, assign it back to autobot to do as much of the build as it can. * When you reach the QA step (Note: Not all builds will include this step), get someone else to assign to QA, fill their name in the box and have them complete the QA section. * When the build is complete, change Build Status to 'Completed' === QA instructions === * As a QA drone it is your job to ensure that the machine is in a usable state and that the build matches the requirements and all steps were completed. * Check the draft build email contains the correct information. * Tick the Final QA checkbox * Assign the build back to the builder and notify them.