Engineering Workstream Renewal, February - May 2023

Proposal Discussion
1

Engineering Workstream Renewal, February - May 2023

Abstract / Overview

The ShapeShift DAO relies on the Engineering Workstream to maintain the open source code base, undertake core engineering work to enable new features, support integrations from the broader community, and to provide architectural oversight and technical leadership as we continue to execute the DAO’s vision.

This proposal is to fund the Engineering Workstream’s budget from February 1st, 2023 through May 31st, 2023 inclusive.

This proposal adopts SCP-92 Definition of Workstream Leader https://snapshot.org/#/shapeshiftdao.eth/proposal/0x3529a9ab550387d1eedf34d0ba26b5c80ed1c04a74a3c407d9b06a6ea90325ab

Motivation

Previous proposal

The Engineering workstream came in under budget of the previously approved proposal

Float of $27k USDC stables still in colony, and cumulative ~400,000 FOX in Colony/operations wallets.

Completed

  • Multi account support

  • Account caching and failure tolerance and major performance improvements

  • THORChain swaps integration

  • Integration of Idle finance

  • Sunset of beta.shapeshift.com and legacy mobile app

  • Sunset of fox.shapeshift.com with LPing and farming functionality now native in the app

  • Swapper v2 with 0% THORChain failures

  • Launch of new mobile app with assistance from Fox Foundation

  • Revamp of fiat ramp UX with new dedicated page

  • ETH/FOX Farming V5

  • Implementation of best rates for swap routing

  • DeFi/liquidity providing abstraction

  • EVM abstractions upstreamed into Blockbook

  • Eligible opportunity UI

  • Migration of infrastructure from Fox Foundation to DAO

  • Self hosted Cosmos and Avalanche nodes, removing reliance on Datahub and improving stability

  • Additional Turkish and Ukrainian language support

In Progress

  • Support of Osmosis bounty & integration

  • Wherever notifications (EPNS integration)

  • Optimism integration (eligible for ~$250k possible grant funding)

In QA

  • WalletConnect to dapps

Team

Mandate

This proposal seeks to have the Engineering Workstream be the sole maintainers with discretion over CODEOWNERS, permissions and administrative rights of the ShapeShift GitHub repository. The billing account shall be controlled by the Fox Foundation.

This is to ensure the DAO maintains a high standard of code quality, velocity, reduced regressions, and has a single core team acting as maintainers responsible for the codebase.

The Engineering Workstream will work with external contributors to ensure contributions can be made expeditiously while not compromising quality, patterns or stability.

Budget spreadsheet

The details below expand upon the budget spreadsheet https://docs.google.com/spreadsheets/d/1vOpFwELxFk2vKBGa9GxiQEo3gDGDReJkZVKk3KvMSwE

Recurring costs

Labor

  • Any contributor reserves the right to take any portion of salary in FOX and locked varieties thereof.

  • The budget includes provision for two new open positions at a rate of up to $210,000 each, denominated in USDC.

Infrastructure

The Engineering Workstream pays a managed-services provider, TaxiStake, for infrastructure that powers the entire backend of the open source platform and mobile app.

Infrastructure costs exceeded our forecasts in the previous budget cycle, due to multiple factors.

  • A migration of the development cluster from the Fox Foundation to under the DAO’s control, via TaxiStake. The development cluster is necessary for addition of new nodes/chains, local and hosted development environments, upgrades, rolling out breaking changes with blue/green deployments etc. This is another progressive decentralization step, where the costs of the development cluster were hidden and previously absorbed by the Fox Foundation.

  • Addition of archival nodes to the infrastructure. Within this budget cycle, this will allow us to remove reliance on Etherscan for internal transaction data required for balance and transaction history, and Infura for archival node data.

  • Node snapshots of synced or syncing nodes - cloud storage costs are expensive.

  • Additional EVM chains, namely Optimism - EVM nodes are expensive by virtue of storage and data.

The following factors are expected to increase the infrastructure costs slightly during the next budget cycle:

  • the addition of more EVM chains, namely Binance Smart Chain and Polygon, requiring archival node syncs, and multiple snapshots, and data ingress.

The following factors are expected to decrease the infrastructure costs slightly during the next budget cycle:

  • Negotiated lower margin with TaxiStake. Previously a 50% markup was applied to the underlying infrastructure costs, this has been reduced to 30%.

  • Removal of EVM full nodes (Ethereum/Avalanche/Optimism), once archival nodes are fully synced.

  • Removal of outdated snapshot data for currently syncing nodes.

  • Migration of API layer from eu-west-1 to us-east-1, reducing application layer costs by 50%, though this may not be worth the labor cost to enact.

TaxiStake is currently forecasting a bottom line cost of $22,100 for February.

The numerous factors above make an accurate forecast difficult. Overall, we are expecting a slight net increase in costs, and are requesting $120,000 USDC for expected infrastructure costs, based on an expected maximum $30,000 for four months. Failure to cover these costs will lead to a 100% production outage.

Non-recurring costs

Bounty Program

As discussed and largely agreed within the community, the bounty program in its previous form had significant flaws. We are requesting $8,000/month worth of FOX, ~200,000 FOX at current prices, to provision for further experiments with tightly controlled scopes of work that are timely and aligned with the shared roadmap where possible.

Miscellaneous

The budget also contains a line item for $5,000 USDC of discretionary funds which may be used for travel, conferences, retreats etc, at engineering’s discretion.

The line item above is provisional, and any unspent funds will be returned to the DAO.

This budget seeks approval for the commitment of $5,000 USDC and $32,000 worth of FOX for non-recurring costs.

Contingency

Unforeseen costs will arise, and 10% of the USDC subtotal is requested accordingly.

The budget seeks approval for $52,800 for contingency to be spent judiciously at the workstream’s discretion, shall it be required.

Variance

Unused funds from any budget category will be returned to the DAO at the end of the budget cycle.

In the event that the Engineering Workstream requires more funds, a separate governance proposal shall be raised.

Deliverables

Goals

The Engineering Workstream continues on in spirit from the previously approved proposal(s).

The workstream exists to execute on the shared DAO Roadmap Proposal SCP-121: Roadmap Proposal Jan - Mar 31, 2023 https://snapshot.org/#/shapeshiftdao.eth/proposal/0x5f7f4b731874f6b1d32405c0842223687c32625fb99064ab0a5bbc552563bed1

The DAO recognizes that the crypto ecosystem is rapidly evolving and liable to change quickly. The Engineering Workstream reserves the right to add, remove, or reprioritize goals from this list, with the intent to deliver the most valuable and relevant items expediently to the market.

The product roadmap, externally posted bounties, emerging partnerships, and changes in resourcing are all factors liable to change the scope or prioritization of listed goals, however, broadly speaking, we intend to complete the strategic goals that are currently defined.

Reporting

The Engineering Workstream will report progress against these goals via the following methods

  • availability via Discord, covering most time zones

  • weekly product/engineering public meetings

  • weekly governance calls

  • on demand for any specific deliverable

Should the community request other reasonable forms of reporting they will be considered on their merits.

Benefits

Fundamentally, the DAO delivers a product to increase ease of use of the multi-chain multi-wallet multi-protocol universe that we find ourselves in. It is the Engineering effort that builds our product. We need Engineering in order to thrive. The goals listed above will be delivered.

Drawbacks

Engineering labor is the most expensive component of the DAO’s budget. Any aspect that does not seem appropriate to serve the greater needs of the DAO should be criticized.

Vote

FOR: Fund the Engineering Workstream from February 1 2023 to May 31st 2023 for $580,800 USDC and $32,000 worth of FOX, with 0xdef1cafe as the Workstream Leader.

AGAINST: Do not fund the Engineering Workstream beyond January 31st 2023.

2

I think we should allocate more funds to help increase the output for engineering and allow for more spending and bring in more quality contributions and contributors. Not to say anything negative about our current team, I believe we need accelorate the “buidling” and get to a product market fit while our timeline allows.

I have full faith in our current code team, but would love to see some additions to increase to the velocity we “release” features.

I support this proposal with changes of adding additional contributors.

3

Thanks for the support PTT, and I agree with you. I think scaling up in this proposal is premature - the SMART goals meeting today being evidence of that.

We still need better alignment with a single, time-phased, resourced plan, clarifying/eliminating goals that overlap or conflict with each other, before we allocate more capital towards scaling engineering.

Off the top of my head we have at least three overlapping plans, and we need one - a passed proposal linked above, a “150k MAU” goal, a $1m grant money goal.

4

Let’s get the renewal through as an order of business, get more clarity and alignment, then go harder.

5

Can we see a comparison of the engineer salaries from last budget cycle versus this one and an explanation for any increases or decreases in salaries?

6

The DAO’s engineering team is one of its greatest strengths. It’s amazing to me how much it gets done with the existing group. I’m 100% in support of defi’s proposal (and leadership) and look forward to seeing what else the team can build in the next two quarters.

That said, I do share the sentiment voiced by @PTT that we might build even more if there were more engineering resources. For instance, we could iterate faster, plow through the roadmap more quickly, and/or experiment with new initiatives in a lean-startup, fail-fast context. This last ability is especially important, IMHO, given the DAO’s continued search for a product/market fit. Perhaps a larger team would provide the ability to create and build entirely new features, while also allowing the roadmap to be implemented in a timely fashion. Bear markets can be great for running experiments and trying new things.

But yea - that’s a discussion for later. For now it’s good to see a clear proposal articulated for Q1/Q2.

7

No change.

8

Glad to see the work stream continuing on!

@0xApotheosis @gomes @0xdef1cafe and kaladinlight have all been amazingly dedicated contributors

I too just want to echo the sentiment around adding more resources to the team. I understand that the DAO still needs to reach consensus on goals and knowing how slowly the DAO moves, it will take time to happen. I wouldn’t want to add more time after that for another governance cycle.

If you had the budget for 2+ additional engineers, it doesn’t require that you scale the team today, but it does give you the ability to do so much quicker when you feel that we have alignment on a plan.

9

I like this idea a lot, 0xean. That doesn’t mean the dao needs to spend those dollars right away, but are available when you are prepared to scale.

10

If the DAO would like to ‘stay in the game’ for as long as possible, retaining and growing a highly skilled and passionate Engineering Workstream is vital and of the upmost importance to accomplishing it. We’re a software product before we are anything else.

The velocity this team has introduced and hardened into culture at the DAO has seen the amount of releases increase from once a week, to over 100 releases since beta migration (<4 months + holiday.) This team has been hellbent on sharpening focus while consistently producing high quality deliverables and maintaining stable development environments throughout.

Over the evolutions of workstreams and contributors since the DAO’s inception, the current proposed Engineering Workstream team has been at the forefront and initial catalyst for dreaming and achieving more DAOwide in output, expectations, standards, quality, processes, and accountability. Keeping that focus and determination steadfast (and potentially growing) in this workstream is crucial to the DAO defining and succeeding with a product that meets market fit and truly is a master at one of the current strategies or services ShapeShift is currently only ‘good’ at.

I agree with the comments seeking additional budget for scaling the team size, but also agree that the scaling should come after more clear vision and strategy are outlined and queued up. @0xean’s suggestion that the budget for future workstream members be added but discretionary seems like a good solve to remove the need for another proposal for budget increase come the time of scaling need.

The increase in budget due to the transition of the cluster from the Foundation to the DAO make sense and the cost saving measures outlined in the proposal to counteract and decrease this additional spend are very appreciated.

Thank you @0xdef1cafe for getting this proposal drafted and keeping the wheels spinning.

11

@PTT,

It is undeniable that we need more engineering resources available in order to accomplish the objectives that we have defined in the time we have remaining. With that being said, I cannot support allocating additional resources and responsibility to the current engineering leadership.

From one of the conversations that took place at the DAO WOW - it is the responsibility of workstream leaders to exercise their best judgment in making decisions on matters within their purview; it is the responsibility of FOX token holders to evaluate the quality of that judgment and vote accordingly.

To summarize the current situation that the DAO is in with respect to engineering resource allocation:

  • Engineering is currently severely underresourced for the task at hand.
  • The DAO does not have the funding available to quickly remediate the engineering staff shortage without cannibalizing resources from other workstreams.
  • The current engineering staff shortage is exclusively the result of decisions made by engineering leadership before the last budget cycle.
  • This outcome was inevitable given the information available at the time that the decision was made, and the current predicament was completely avoidable.

For context, the decision to cut 55% of engineering staff last year was said to be primarily motivated by budgetary concerns. This is fair, as the project is necessarily dead in the water if the treasury reserves are depleted. Engineering was already said to be underresourced at that time, so there was obvious concern about cutting the available resources further.

To simultaneously satisfy the concerns about budget and resourcing, multiple strategies were proposed that would have minimized the detriment to engineering capacity while providing nearly the same cost savings as removing 5/9 engineering positions. These strategies were not pursued for reasons that seemed nonsensical to many at the time, and the outcome of those decisions now speaks for itself.

To anyone who fully understands the technical lift required to accomplish our mission as currently defined, it was never even a remote possibility that this could have been done in the timeframe allowed by the runway with ~3.5 full-time engineers. Furthermore, the engineers that were removed from the team were each long-tenured employees from centralized ShapeShift, talented contributors with broad industry context and domain-relevant expertise, and had incentives that directly aligned their interests with the interests of the organization. Many of those contributors were willing to take 100% FOX compensation with much of it time-locked, specifically out of interest in maximizing the DAO’s chance of long-term success. Contributors like that are very difficult to replace. All of this was known at the time, and the quality of judgment exercised given the information available was exceedingly poor.

I don’t believe there to be a great deal of utility in continually harping on previous failures; however, without complete recognition and acknowledgment of those failures, it is impossible to ensure that one is likely to do better going forward. I am deeply concerned about the lack of accountability that I have seen from engineering leadership for the current predicament that the organization finds itself in. This is in line with a lack of accountability that I have seen from current engineering leadership regarding managerial failures on the previous team. Some at the DAO have stated their belief that individuals are generally incorrigible and that every action taken by an individual invariably speaks toward the character of that person fundamentally. I do not share that belief, but I am very much of the opinion that individuals typically do not improve until they are able to take personal responsibility for their previous mistakes.

Above all, the objective here is to maximize engineering output however possible. It has been said many times that the configuration of the current engineering team is optimal, that the number of engineering contributors on the current team is near the upper bound on the number of engineers that can be managed effectively by one person, and that team cohesion and efficiency are at an all-time high for this working group. If that is the case and the current arrangement works for that group, then we should keep that going. From what I have seen in GitHub contribution statistics, the three contributors working under current leadership are producing high-quality output as quickly as possible and delivering value to the DAO and to our users. We should empower them to continue the good work.

Still, the fact remains that we need additional resources available. As much as I wish this was not the case, I cannot at this time give current engineering leadership a positive performance review overall, and thus cannot justify an expansion of budget or responsibility. To responsibly expand the engineering capacity of the DAO, I recommend establishing a separately managed team, functionally equivalent to and operating in parallel with the existing team. The two teams will act cooperatively to jointly accomplish the shared vision and will take separate sets of features from the product roadmap so that each team is able to operate under a narrow focus with a minimal number of items simultaneously in flight. This option best satisfies the current concerns outlined and establishes a model that will allow us to scale effectively going forward.

12

I support the proposal as it is written. I would not support a proposal that increases the number of engineers in this workstream. We must break this culture of centralizing all engineering efforts to a single team. More engineers under the direct management of a single workstream leader will not speed up or increase the productivity of this team.

9 Women Can’t Make a Baby in a Month

-Fred Brooks

We need more engineers working for the DAO; and need to promote a more open and flat organizational structure—the centralized old-school thinking of managers and “reporting to” leadership is holding us back and slowing us down.

We should foster an engineering culture, hire more engineers, and work with them outside of a single manager/leadership role. We are underestimating the power of experienced engineers leading efforts for the DAO independently, and a large amount of talent is floating around in orbit. The DAO would be wise to capture and manage outside of a single workstream leader.

13

Just for posterity and to allow for more time to address if desired. I will post my questions and the answers I received from the last governance call on 1/26/2023. I will provide some more context as well in this medium with time constraints removed.

As currently listed the budget shows $32000 FOX all non recurring, so no contributions paid-out in FOX?

This was the question, the answer provided was it is up to the individuals to determine what they want to do and therefore not needed.

Follow up questions:

  1. Most of the other workstream contributors have elected to take significant portions of their compensation in FOX in order to help the DAO minimize its USDC spend as we hunker down during difficult financial times. As a workstream leader and the highest prioritized workstream, why have you specifically chosen not to do the same? Or offer good will to the voters of the community?
  2. If you can't speak to others determination, can you as a proposed workstream leader share yours? Leading by example

Question from call; Last proposal was 5 months and this is 4 for the same recurring compensation amounts and calculations, was the last budget incorrect? Answer: I was nervous, this was a miscalculation or error and no one discovered it. (Paraphrase) No foul

Question: If you bring on new engineers as part of this proposal, it’s going to take them some time to get onboarded and the current engineers are going to have to do the onboarding. As a result, engineering output can be expected to decrease over the duration of this proposal. How do you plan to make that work with the DAO’s general strategy, limited runway, and within limited engineering capacity?

Answer: How do you know, what is your experience? (lists two current devs and the only new full time contributors for the workstream) I provided two examples that prove it to be Immediate and your assumptions are incorrect.

As stated, that is a false equivalency, and misleading. Those devs were onboarded and spent time as bounty hunters getting familiarized and helped within the code base. Yes they are talented and hard working, no knock on that, but to say immediate is a mischaracterization and incorrect, also a very small sample of experience to say that is the norm.

Question from call; Why are you paying contributors different amounts for the same role?

Answer: Different skill, tenure, and experience.

Lastly I pointed out that budget links don’t work in the post without manual edit of links, if you wanted to fix that.

I know that these are not easy questions and I hope that others are also willing to ask the hard questions moving forward, but more importantly that we have open community communication and don’t ignore those who are willing to ask.

14

I pointed this out. your merging 2 different things. the $32k is for expense payouts in fox (and $5k in usdc). nothing to do with individual fox. Individual fox its a choice that each contributor makes. there is no forced option in place. A few (kent) put it in their proposal. I did at one point, but only as a measure to try it out. after that, we choose what our % is. Last month i (january) i chose to do 20%, as i needed the funds. but im ok for a bit now, and already set my fox allocation to 100%, I have outside funds however. I wouldnt expect anyone else to do such.

15

@Neverwas, you brought up some good points during the governance call today. Thanks for posting the follow-up here!

@0xdef1cafe, being that this proposal includes a request for treasury funds, I think it’s well within the rights of the FOX holders to have any questions or concerns about this proposal answered or addressed at this stage of the governance process. As a FOX holder, I would very much like to see Neverwas’ questions answered before moving this proposal to the ideation phase.

Additionally, I would like to hear why you think it is reasonable to request that the DAO expand both your budget and responsibility, specifically given your many previous arguments on why it would be directly harmful to engineering output if your team was expanded, and in consideration of the historical context and concerns I’ve discussed in detail in my previous post.

I look forward to continuing the conversation here prior to moving this through the remainder of the governance process.

16

After discussing within the team, and considering the valuable comments from @seven7hwave @PTT @0xean @TylerShapeShift the original post has been updated to include two additional open positions.

17
  • To answer your questions

    Personal financial decision.

  • I can't speak others decisions. See above.
    • 18

    Love this update! Thanks for taking constructive criticisms and making adjustments as people had said!

    19

    Moved to ideation https://snapshot.org/#/shapeshiftdao.eth/proposal/0x0d38e777ed469bf86fa2004113424e4e2f1e011633480c64ff80f3f52665e181

    20

    This Ideation proposal does not respect our current Governance process, it is missing an option:

    Must include three options:

    • For/Yes with no amendments/changes
    • For/Yes with amendments/changes
    • Against/No