DevCycle | Senior/Staff Full Stack | Full-time | Toronto (Hybrid / Remote) | $150k - $180k | https://devcycle.com
We are looking for a Senior-level engineer to help us build out and scale DevCycle. We are a small team that's been building in the feature flagging / AB testing space for a while (from Taplytics YC W14), supporting open standards like OpenFeature. We run a large distributed system serving trillions of monthly evaluations, relying on technologies like Cloudflare Workers, Next.js, Nest.js, WASM, etc.
Also, once you are on Enterprise, they will not bug/charge you for contracted overages very often (like once a year) and will forgive significant overages if you resolve them quickly, in my experience.
Actually, M4 is reportedly on a more cost-efficient TSMC N3E node, where Apple was apparently the only customer on the more expensive TSMC N3B node; I'd expect Apple to move away from M3 to M4 very quickly for all their products.
Owner of the purposefully controversial opinion here :) Looks like its proven to be very controversial, which was the stated goal of the podcast segment :)
For context, I had just returned from KubeCon + CloudNativeCon EU in Amsterdam. I had a chance to talk to many of the great folks pushing the WASM ecosystem forward, and there certainly seemed to be a large amount of momentum behind using WASM in the server space and at the edge. I'd had a great time learning from the folks at various open-source projects / companies pushing WASM forward:
At DevCycle we are using WASM to help share core Feature Flag decisioning code across our Cloudflare Worker APIs and our server SDKs for local decisioning. We've learned a bunch about compatibility / optimizing performance / limitations that I'm happy to chat about in more detail. But we are generally excited for the future of WASM, especially for the new features coming: Component Model / Threading / Garbage Collection changes / Networking.
If you are looking for how to use WASM Components to replace container-based runtimes, look at the companies linked above.
Happy to provide more details on this. In most cases, our server SDKs use a common WASM library to make all user bucketing and segmentation decisions locally. We have been impressed with the performance we've achieved with WASM. It allows us to share our most critical non-trivial amount of decision-making code, and it lets us share it between our Edge APIs and Server SDKs. So the same WASM code making decisions locally in our server SDKs is also making decisions at the edge for our APIs powering our client-side SDKs and public APIs.
We have stuck to the Bytecode alliance-supported Wasmtime runtimes that we highly recommend for anyone going down a similar path.
There have been cases with highly parallel environments where we needed to support micro-second level optimizations for our GO SDK, where we ended up building a native version of our bucketing logic. We hope that WASMs' upcoming threading and memory management improvements will make this use case supportable without needing native code. But for our higher-level language SDKs like NodeJS / Python / Ruby / PHP we've generally seen as good or better execution performance from WASM.
What desktop environment are you looking to support? We have customers using our iOS SDK in MacOS (we need to find a better name for our Apple platforms SDK that supports iOS / iPadOS / MacOS / WatchOS / tvOS). Our React SDK works with electron apps, and we have on our roadmap to investigate client SDK support for our .NET SDK.
Really interesting take on your experience with experimentation / feature flags and wanting it to be more stateful in code. I assume that follows the more Terraform-like infrastructure as code approach, which certainly makes sense to rollout infrastructure changes. We have a V1 of a Terraform provider where we hope to enable more control through terraform to manage infrastructure changes directly, but you are correct that has yet to be our core focus. Another way we hope to expose that state outside of our UI is building out a great CLI, which we are actively working on.
Our primary usecase has been working with product development teams who have adopted Feature Flags as part of their workflows, using DevCycle to help them deploy features faster / safer to their end users, even deploying continuously as we do. Generally, customers integrate DevCycle into their Websites / Mobile Apps + API servers to control those feature deployments. We have had a couple of customers use us at the infrastructure layer. One of the best use cases I've seen is in a proxy service controlling the rollout of a new infrastructure stack. I'd love to dig into this use-case deeper and see how we can better support it in the future, "Ops" flags are one of the flag types we are looking to support.
I would say that for customers looking to disconnect the deployment of code from the release of features, our approach where you can "release" features at any time has many advantages over "git-ops" style configurations. But certainly, for Ops use-cases where you are controlling infrastructure changes, we are believers in "git-ops" and use it ourselves. The challenge comes with how to connect those two deployment styles effectively.
Thanks for the detailed reply, although I feel like I may have miscommunicated the aim here. I'm not really thinking about what is being launched, but how, and where the truth lives.
A CLI is certainly a nice feature, but brings decisions that need to be made: who runs it, where do they run it, when, how do you know what was run, how do you deploy the CLI, and so on. The same can be said for, say, running your test suite – and the solution there is to have CI do it for you. Sure you can run, but the run that matters is the one when you merge your branch and that's done in a controlled environment typically defined in code.
Rolling out features/flags is the same, and I think if the state of all the flags, features, traffic allocations, targeting, and so on, is all defined in code, then all those questions you get with a CLI go away. Who runs it? An automated process. Where do they run it? In a controlled environment, not on a dev machine. How do you know what was run? It's all there in code. How do you deploy the CLI? You don't need to.
> for customers looking to disconnect the deployment of code from the release of features, our approach where you can "release" features at any time has many advantages over "git-ops" style configurations
I'm interested as to why you say this has many advantages, because I don't see why a git based workflow couldn't run this. You could for example have DevCycle subscribe to the notifications for the git repo and update its internal state any time new changes are made to the git repo. That would be preferable over a UI or CLI because the whole state is there in a machine readable format ready for tooling to use.
You can use a git-based workflow for feature flags, that's likely how most teams will start using flags in their code with environment variables and infrastructure state. However, most deployment pipelines in the wild are very slow and owned by engineering, limiting the value of doing git-based state for your Feature Flags.
I've seen that disconnecting the deployment of code from releasing of features can be transformative in various ways that fit better with an API-based model for Feature Flags:
- Many flags / remote config changes in production environments are made by non-developer members of a team.
- Coordinating releases across multiple platforms that have different deployment cycles. For example, deploying a new feature to Mobile + Web + APIs simultaneously.
- Enabling / disabling flags in real-time across your stack to respond to incidents.
- Support / Sales teams using Flags to gate features for customers.
- Remote config to populate data for UIs, to act as a CDN for content.
- Rolling out infrastructure changes, being able to roll back changes instantly without another deployment.
(And for context: our Feature Flag decisioning is done locally for our local bucketing server SDKs)
I put this in another comment, but disconnecting of code and feature release can still be achieved while storing feature flags as code. You need to build out a seperate pipeline and release process. With this you can still achieve all the advantages that you listed out.
I think the key difference is if you consider feature flags to be configuration and believe in configuration as code as a best practice.
I think this is the right approach in the long term. People having the ability to modify configuration in production in real time is an outage waiting to happen without having an obvious way to rollback to previously known working state.
We certainly see workflows where defining everything as code can be powerful. As we've been chatting about this internally today, we have ideas of how you could define all your features / audiences / variables as code and only deploy changes using a Terraform interface.
I feel like, ultimately, this overly limits the reach of feature flags within an organization; it makes it more difficult for most engineers who are not comfortable with Terraform to deploy their features; it limits the ability of product / marketing / project managers / QA from; scheduling releases, configuring experiments, testing features, modifying audiences, etc.
Modifying configurations certainly can cause production issues, which is why we have built tools like our variable schemas and strong typings for our variables to limit the value changes to only known values. With our CI / CLI / IDE integrations, we make it known what values variables can be set to, when variables are added and removed, and when variables enabled in production are modified by PRs. This is all in addition to existing permissions / change logs / rollback functionality we are improving on. One of our core responsibilities is to make changing Feature Flag values as safe as possible. We know that flags can quickly become tech debt, so we also preach that feature flags should be as easy to remove from your code as they are to add to your code.
A fully featured feature management platform should be with you from your IDE / CLI, Code Review / CI, Deployment Pipelines / Alerting and Monitoring, and integrated with your Project Management and Product Analytics. It should be visible to your team throughout their workflows what features are enabled across all your environments.
I don't think using source control is incompatible with this. You can persist state in source control, and have a separate deployment pipeline that is triggered only for updates to feature flag files. It is also deployed separately; e.g., to a key-value store.
Prefab has something that you could use in more or less this way. The value of the flag can come from different sources and there is a defined order precedence. The base levels are default yaml files, which would be committed in your project. We think of live values from the server as a "potential override". So if you only wanted things in git you could just not use the UI. Details in https://docs.prefab.cloud/docs/explanations/bootstrapping
I'm spike-ing on some similar/adjacent things right now actually if you want to jam. I won't claim that what exists today is very polished for your usage pattern.
(DevCycle Co-Founder & CTO here) The truth of the Feature Flagging market is the vast majority of engineering teams aren't using a third-party tool for Feature Flagging. Most teams are using an internal system (Environment Variables, Database flags, S3 files ect.) or not doing any feature flagging. The potential for the Feature Flagging market is as large as any other developer tools market, and the challenge is to build a product with a better developer experience that encourages the adoption of feature flags and integrates with team's workflows.
We've been in the A/B + Feature flagging space for about ten years with our visual A/B testing-focused product Taplytics (W14), so I'd say we have lots of experience running Feature Flags + A/B tests at scale. We've learned a lot from our customers over the years and were excited to build DevCycle as a green-field project from those lessons. If you are interested, try it out and we are always open to direct feedback.
Interesting that the differential between different regions is remaining the same.
You would imagine that if some big user were slurping up all the spot instances worldwide, then it would level out regional pricing differences. Or, if not level it out, change it based on that users sensitivity to bandwidth costs in/out of each region.
That makes me think that the spot pricing change is a gradual AWS algorithm change instead - I'm sure they'd be smart enough to apply a pricing change with some smooth function based on date.
You can shop around a bit for AWS for spot instances. Spot prices in eu-west-3 (Paris) are not that wild yet for the same c5a instances. Somteimes, a region has all its spot instances near the on-demand price. At least they seem to be smoothing out their changes over days. (I think it used to be more wild in the past.)
The problem is there's been growing demand across the world and/or more people optimizing costs and moving from on-demand -> spot. It's not a single user doing this.
There were hardware shortages earlier. I think it's just a case of AWS can't expand fast enough to meet demand for now.
Important to note that AWS doesn’t control spot prices: it’s a market where you bid on what you’re willing to pay. Customers control the cost through their bids.
edit; and prices tend to surge at end of quarter, regardless of provider. I’ve seen that for years at AWS and it’s only increased year over year.
But Stocks and Currencies aren't technically a digital good, they are partial ownership of a company or a hard currency. A NFT is exactly the definition of a digital only good, seeing as how a large number of NFT use-cases are to replace / make portable in-app goods sold in online games / mobile apps, which all currently require IAP fees to purchase on mobile. So this is just Apple applying the same rules to NFTs that apply to purchasing items in mobile games.
Apple was never going to allow the blockchain to remove their 30% cut on these purely digital items, as much as crypto folks preached that the blockchain would remove us from the shackles of App Store IAP payments.
We are looking for a Senior-level engineer to help us build out and scale DevCycle. We are a small team that's been building in the feature flagging / AB testing space for a while (from Taplytics YC W14), supporting open standards like OpenFeature. We run a large distributed system serving trillions of monthly evaluations, relying on technologies like Cloudflare Workers, Next.js, Nest.js, WASM, etc.
If you are interested, please connect at: https://jobs.lever.co/devcycle/c985bdfb-c6c4-4640-96c2-da3ec...