Summary This proposal is for a continuation of budget funding for another 6 month term, February 1-July 31st, 2023 (Fourth Term) for the current Operations workstream that was established with continual renewal of the passings: [SCP-8], [SCP-59], and [SCP-94].
Abstract The Operations workstream is an integral part of the growth and development of healthy systems for reliably producing and maintaining the ShapeShift product suite and its uptime, responsiveness and feature functionality. The workstream additionally provides similar support and structure to the DAO, its ecosystem and networks.
Through Production Issue Procedure Best Practices, Triage and Bounty Management, Deployment and Regression Testing, and DAO Admin Procedures and Practices, the ShapeShift DAO Operations Workstream provides essential resources and tools for efficiency, organization, and overall product and services health that are vital to the success of the DAO, the product suite, it’s contributors, and $FOX token utility.
Motivation Renewing this workstream will allow Operations to continue to work side by side with other workstreams to increase efficiency in individual procedural processes across the DAO and help progress deployment and testing processes while still verifying and maintaining reliability, health, and full functionality of all provided services. New practices developed to expedite efficiency through collaboration and administration of procedures over the last term include newly developed responsibilities such as: Marketing content reviews, RDP disclosure management, Partnership flow creation and maintenance, Gitbook documentation management, and full coverage of customer support, among others.
These procedures and collaborations allow the Engineering workstream to spend more time building and deploying; the Product workstream to spend more time scoping, designing, and implementing; the C-Mods more C-Mod time, etc… Funding this workstream empowers Operations to continue to complete the above stated goals through our continually developing practices, and organize and provide updates, resources, communication, coordination, documentation and any other needs that may arise from other workstreams, the DAO Community and its processes.
The Operations workstream is the only workstream focused on the wider picture of the product suite’s uptime, functionality, and overall health. It additionally mirrors that responsibility in the DAO Discord, Github, and Notion by synergizing and coordinating cross-stream collaboration further strengthening the DAO.
Specification The Operations workstream Notion repo can be found here.
Operations maintains: Triage tracker and Triage Report Tracker The ShapeShift Dework Bounties Board, Trusted Community Dev timesheets, Bug Bounty Tracker 2022 Prod Issues, Release Stats and testing fees, Regression Testing Schedule Ops Review in the Content Production Roadmap Ops Sprint, Governance Call, AllFox, Go/NoGos, Retros RDP program, Gitbook, Customer support in Discord
The Operations Workstream strives to keep the greater DAO agile and committed to maximizing efficiency, scalability, and transparency with a strategic lean into data-based decision making. At times, best practices and procedures may be challenged, modified, updated, changed, archived or revisited to maintain those objectives. (See: Unsolved User Problems, the governance tracker, POAP minting, daily regression testing, the evolution of goat/no goats, etc)
The renewal term length for this workstream and its contributors is 6 months, from February 1st 2023 to July 31st 2023.
This proposal continues @Tyler | ShapeShift as the workstream leader and @Tshifty, @Ron/PTT, @MBMaria, and @wesleygraves as Operations Assistants.
Budget ask: $262,500 - 20.45% less than the last term
Monthly USDC: $32,425 - 17.5% reduction from $39,275.00 last term. Monthly FOX: $11,325 - 28% reduction from $15,725.00 last term. Monthly total budget: $43,750.00 - 20.45% reduction from $55,000 last term.
6 month USDC budget: $194,550 6 month FOX budget: $67,950 6 month Total budget: $262,500
Full budget details
A few changes have been made from the previous term in this next proposal with regards to staffing and budget reductions. As the bear market and DAO buidling march forward, the overall demand of Operational administration and testing requirements has evolved and reduced. Regression tests have gone from daily to a monthly rotation. Deployment test procedures have reduced the amount of time and contributors needed to efficiently respond to deployment testing requirements. The bounty program has been chilled until further strategic conversations can redirect it to a more successful and efficient workflow. Operations bases its focus on efficiency and in the spirit of optimization, it has archived and reduced cross workstream dependency on procedures and meetings like goat/no goats and Unsolved User Problems.
Across the board, these optimizations have pressed hard against the need for the complete staffing proposed and fulfilled in the previous term as months progressed and the DAO shrunk and refined. Even the inclusion of all customer support responsibilities did not move the needle away from the clear sign that the Operations workstream should not be exempt from the hard staffing decisions that had been made in every other workstream to this point.
Contributors @Neverwas, and @lych have both made quality contributions to the DAO over their tenure with the DAO Operations Workstream. They have been given 1 month severance, sent from Operations excess budget in Colony to help with this transition. Each contributor was informed of this decision in early January and was additionally paid for the rest of the month without expectation of any further responsibilities to the workstream.
Tyler and the current Operations Workstream are confident that the goals and responsibilities of the workstream can be proficiently maintained with a smaller and more focused team.
The budget for testing funds has been reduced $2,500 USDC this term to match testing requirements.
Tyler’s Workstream Leader salary has been adjusted from $12,500/month to 15,000/month and the % FOX has also been reduced in the WS leader salary from 40% to 15% this term.This adjustment aligns the WS leader salary with the lowest paid full time Product Workstream contributor passed in proposal [SCP-119] This workstream salary has not seen any changes or increase in the previous 14 months of service, while Tyler’s contributions and responsibilities have greatly increased. Tyler forwent any salary adjustments last term to assure the pay raises given to all other Operations contributors then.
Operations accounted for 11% of the total monthly DAO spend in the last proposal. The DAO is lighter by a number of Workstreams this term so the potential % of the total DAO monthly spend for this term’s budget is 16%.
If network conditions or the price of FOX decreasing dramatically alters or changes Operations ability to perform required deployment and regression testing, an additional proposal addendum for test funds may be proposed.
Specific Goals and Success Metrics The goals and success metrics for the Operations Workstream are listed below, split into current Goals and KPIs.
Current Measurables:
Production Issue Best Practices
Goals: To minimize downtime of the entire product suite. To resolve production environment bugs efficiently with minimal disruption to engineering processes, deployment cadences, or user experience.
KPIs: 99% uptime of the product suite and services: July: 99.66% August: 100% September 96.6% October: 93.2% November: 99.8% December: 98.7% Average 6 month uptime: 97.9% <8 Production Issue Escalations per Quarter - 6 Escalations in Q4, 2 Escalations in Q3 <50 Total Combined DAO Contributor hours dedicated to Production Issue Resolutions Per Quarter (Q3 - 24.5, Q4 - 23 hours)
Triage Management
Goals: To discover, isolate, navigate, and manage resolutions of user facing issues in all ShapeShift products and services. To provide known troubleshooting and mitigation steps for issue reporters and users during the discovery and recreation of triage tickets. To mitigate and expedite time required from core contributors spent on bug tickets, issue reports, and production level emergencies. To transfer correctly completed and accurate Github tickets of triage bugs to Engineering or Product to be posted for bounty.
KPIs: Greater than 80% resolution rate goal: 66% resolution rate this year.
This goal was not hit this term as many of the admining and oversight cycles spent on ticket flow management was refined to direct github ticket writing and the loss of the unsolved user problems ticket clean up workflow. This process will be updated and optimized this term as the workflow from support issues to ticket generation/monitoring are now all in one process. This KPI will be updated with the new flow and communicated accordingly in future AllFoxes.
Deployment and Regression Testing
Goals: To catch 95% of prod issues during the deployment testing process. To catch user facing issues in production environments in regression tests and #triage them accordingly. A deployment test is completed by the Operations Workstream during each release process and upon request in pre-deployment branch testing. Specific regression tests are performed once a day by an Operations Workstream contributor to complete a full monthly regression testing schedule in accordance with regression testing best practices. Operations prioritize blocking bugs, supporting engineers, and efficiency in the processes already in place over chasing non-blocking bugs for a ‘perfected’ release.
In the 120 releases created in the newest iteration of the deployment testing process since September, 8 total prod issues were experienced with only 3 of them being caused by bugs that could have been caught in the deployment testing process. 3 bugs in 130 releases = 97.5% This does not include the releases from the earlier version of testing procedures.
KPIs: <2 Cadences missed per quarter due to bugs overlooked in Ops Testing (0 cadence missed last term) <2 Rollbacks during releases per quarter due to bugs overlooked in Ops Testing (0 Rollbacks occurred last term)
DAO Admin Procedures and Practices
Goals: To organize successful DAO meetings related to Goat/No Goat, Retros, Ops Sprints, and AllFoxes. To track, organize, record, and optimize DAO repos keeping them current and relevant. To help in cross workstream collaboration projects that optimize efficiency and collaborative, transparent communication. To provide assistance in projects that require support in generating procedures to be completed for the DAO or other workstreams.
KPIs: 72 hour max response time for all PR review requests from Marketing in the Content Production Roadmap. Keep All Workstream leaders and key contributors informed and involved in contributions to all Ops Sprints, Governance Calls, AllFoxes Slide decks, Go/No Gos, and Retros.
This goal was hit, however the process has matured and does not need the same level of focus in this next term.
Benefits:
If this proposal passes, the DAO will continue to fund the Operations workstream budget to meet the deliverables and services outlined which are integral to the DAO’s success and current practices. Coverage for Prod Issue escalations, Triage Management, Regularly scheduled regression and deployment testing, Cross workstream alignment and stakeholder resources would all be focused, maintained and improved upon by the workstream.
Passing this proposal will continue uninterrupted support and services offered from the Operations workstream to the DAO.
Drawbacks:
Funding this workstream costs the DAO stablecoins and FOX during the duration of the proposed term. This workstream would cost the DAO treasury potentially $262,500 over 6 months (~$43,750/month) if all funds are needed and exhausted in the budget.
Vote:
For: You support funding the Operations Workstream’s anticipated budget for 6 months, totaling a potential $262,500 to the DAO, and continuing the roles, responsibilities, and expectations outlined above in the Operations Workstream during that time.
Against: You do not support continuing to fund the Operations Workstream.