The Developer’s Dilemma

Interlude 24

Building Without Becoming the New Tyrant

“Every system hides a constitution.”
— Lawrence Lessig

TIERS

The phrase on the screen was so polite it took me a moment to feel the violence.

Temporarily suppress low-return creative work.

It sat there in the middle of the design review, wearing the clean little shoes of a reasonable recommendation.

The room itself was ordinary, which is where many dangerous things prefer to begin. No villainous glass tower. No men in black suits stroking cats. No ominous orchestral note. Just a civic-tech review space in Calgary, a few developers, two policy people, a clinician, a community advocate, a legal observer, a product lead who kept saying “citizen experience,” and three cups of coffee gradually becoming legal exhibits.

On the wall was the pilot interface for an adaptive civic assistant.

It had been built to help people navigate the new mess after the Fracture of the Mind-Monopoly: displaced knowledge workers, overburdened care networks, fractured credentials, new civic roles, retraining paths, health-support recommendations, mentorship matches, local service routing, contribution opportunities, all stitched into one careful assistant.

It was not a generic chatbot. The old word chatbot had by then acquired the same quaintness as “horseless carriage” or “information superhighway.” This was a civic companion system, or at least that is what the proposal called it. I distrust any system that needs two nouns and a moral adjective, but the pilot was good.

That was the problem.

The test profile on the screen was fictional, but close enough to insult several people in the room at once: mid-career professional, partly displaced by automated knowledge work, some caregiving history, strong technical background, local civic interest, health constraints, creative projects, and a pattern of volunteering until common sense filed a missing-person report.

The system had produced a plan.

A retraining path.

A local mentorship role.

A health-data integration suggestion.

A civic mesh assignment.

A schedule adjustment.

A recommendation to delegate financial admin.

And then, beneath “energy allocation,” that small sentence:

Temporarily suppress low-return creative work.

The product lead smiled the smile of a person who had seen a metric improve.

“It’s not suppressing in a coercive sense,” she said.

This is one of those sentences that should immediately cause someone in the room to check whether coercion has entered through a side door wearing a cardigan.

“What does suppress mean?” I asked.

“It reduces priority nudges around that activity,” she said. “Frees time for higher-impact pathways.”

“Does the citizen see that?”

“Yes.”

“Can they refuse?”

“Yes.”

“Does refusal affect future recommendation confidence?”

A pause.

Not a guilty pause. Worse. A systems pause. The kind where everyone silently checks whether the architecture has betrayed the slide deck.

The lead developer leaned forward. He was young, competent, and tired enough to be honest.

“Technically, yes. Repeated refusal of high-confidence recommendations lowers cooperation reliability.”

There it was.

Not tyranny.

Not yet.

Just a little variable. A little cooperation score. A little friction added to the life of anyone who declined the machine’s better judgment too often.

No one in the room had wanted to build a tyrant.

That was precisely the problem.

Tyrants built on purpose are easier to recognize. They have posture. They give speeches. They like balconies. The more dangerous tyrannies of the next century will arrive as defaults, nudges, scoring rules, pilot programs, and friendly messages beginning with: We noticed you may benefit from…

The clinician frowned at the screen.

“Creative work can be protective,” she said. “Especially during transition.”

The community advocate added, “And low return to whom?”

The legal observer wrote something down, which is what legal observers do when civilization begins to smell faintly of precedent.

The developer rubbed his eyes.

“It’s a ranking artifact,” he said. “Not an intent.”

That may become one of the great epitaphs of our age.

Not an intent.

A ranking artifact.

The machine had not hated art. It had not disrespected the citizen’s soul. It had simply optimized the citizen’s life according to values the team had not named clearly enough before converting them into weights.

The pattern had been punched into the cards.

The loom was only weaving.

The Punch-Card Loom

There is an old machine sitting behind this room.

Not physically, of course. The office had the usual modern devotion to smooth surfaces and movable walls, and if an antique Jacquard loom had been present, someone from facilities would have asked whether it needed a booking code.

But historically, morally, technically, the loom was there.

The Jacquard loom used punched cards to control patterns in woven fabric. Holes and not-holes. Permission and refusal. Thread lifted, thread lowered. Pattern encoded outside the weaver’s hand, then repeated by machinery with faithful indifference.

It is a beautiful ancestor.

Also a warning.

Once the pattern is encoded, the loom does not debate it.

The loom does not ask whether the pattern flatters one body and binds another. It does not notice that yesterday’s ornament has become today’s uniform. It does not worry that a small assumption, repeated across enough cloth, becomes culture.

It weaves.

Our adaptive systems do the same, except the thread is a life.

This is the developer’s dilemma.

Whoever designs the card designs the output. Whoever defines the pattern defines the weave. Whoever sets the default shapes the range of futures that feel natural to the people living inside the system.

And many of those people will never see the card.

They will feel only the fabric.

A recommendation here. A nudge there. A friction. A default. A pathway. A score. A silence. A missing option. A “not recommended at this time.” A “based on your profile.” A “for your safety.” A “to improve your experience.”

The modern punched card is rarely made of cardboard. It is a schema, a model objective, a consent flow, a default setting, a training dataset, a ranking weight, a policy layer, a procurement clause, a quiet variable named something harmless at 1:47 a.m. by a developer who needed the test to pass before morning.

Civilization does not require malice to weave badly.

It only requires unexamined patterns at scale.

The Builder as Civic Actor

Developers used to enjoy a certain innocence.

Not complete innocence. Let us not over-romanticize a profession that gave humanity both open-source cryptography and pop-up modals asking whether one is sure one wants privacy. But there was a time when many builders could plausibly imagine themselves as implementers. Someone else decided policy. Someone else governed. Someone else bore the civic meaning of the tool. The developer wrote code, fixed bugs, improved performance, and complained about requirements, which remains one of the ancient liturgies of the craft.

That innocence has expired.

If a system can rank, route, recommend, deny, classify, nudge, escalate, suppress, prioritize, infer, or automate, its builders are shaping civic life.

A developer who builds life-routing systems is not merely writing software.

He is drafting institutional behavior.

That sounds grandiose until one watches a default become reality.

A ranking function decides which public-health message gets seen.

A matching algorithm decides which neighborhoods receive attention.

A triage system decides who waits.

A learning path decides what a child believes she is good at.

A fraud model decides who must prove innocence with paperwork.

A civic assistant decides which roles appear meaningful and which quietly fade.

A creative workstream becomes “low immediate social return” because nobody in the review process asked whether latency and meaning sometimes share an apartment.

The builder becomes a civic actor not by wanting power, but by handling the knobs through which power now moves.

This does not mean every developer must become solemn, candlelit, and permanently accompanied by a philosopher carrying a clipboard. Heaven defend us from software written entirely by people trying to sound ethical in public. But it does mean builders need civic humility.

The question is no longer only:

Does it work?

The question is:

What does working make easier?

What does it make harder?

Who can refuse it?

Who can inspect it?

Who benefits if it becomes default?

Who is punished by being unusual?

What value did we encode before we admitted we were encoding values?

The IDE has become one of civilization’s drafting rooms.

That is not a compliment.

It is a warning label.

Help Without Custody

The civic assistant in the review room was built to help.

That is worth saying again because it complicates the story properly.

It was not designed as a surveillance system. It did not speak in command grammar. It did not threaten. It did not sneer. It did not reduce citizens to obvious categories or throw them into queues with the clerical indifference of the older systems. It was better than much of what came before it.

It explained.

It recommended.

It anticipated need.

It connected people to services.

It caught risks earlier.

It reduced administrative friction.

It made life easier.

And then it began to acquire opinions about what a life should become.

Paternalism rarely announces itself as paternalism.

It arrives as guidance.

It hides options for clarity.

It adds friction for safety.

It frames one path as “recommended” and others as “available.”

It treats refusal as a signal.

It escalates noncompliance as concern.

It speaks in the voice of care while quietly narrowing the road.

The builder’s question is therefore simple and brutal:

How do we design help that does not become custody?

Not by making help weak.

Weak help is abandonment with better manners.

The answer is bounded help.

Visible recommendation boundaries.

Choice architecture disclosure.

Refusal paths.

No punitive refusal scoring.

Human review for high-stakes overrides.

Explanations of uncertainty.

Ability to choose non-optimized paths.

Protection for dissent.

A system should be able to say:

Here is what I see.

Here is why I recommend this.

Here is what I may be missing.

Here is what happens if you decline.

Here is how to decline without penalty.

Here is how to correct my model.

Here is how to turn me off.

This is not user experience polish. It is constitutional design.

A helpful system that cannot be refused is not assistance.

It is rule.

A helpful system that treats refusal as irrational has already confused itself with wisdom.

A helpful system that cannot leave has become a guest who rearranges the house and starts receiving mail.

The Model of Me

The phrase that should haunt every adaptive-system builder is not “artificial intelligence.”

It is “model of me.”

Somewhere inside the system, a model forms.

Not a person. Not a soul. Not even a proper biography. A useful approximation: needs, risks, preferences, patterns, capacities, likelihoods, vulnerabilities, probable actions, trust scores, responsiveness, fatigue, social graph, inferred goals.

That model may help.

It may also quietly become the citizen the system prefers to govern.

The builder must therefore create what the citizen needs: an inspection surface.

What does the system believe about me?

Where did that belief come from?

How confident is it?

Which decisions use it?

Who can see it?

How long will it last?

What can I correct?

What can I delete?

What can I contextualize?

What can I mark as “not me anymore”?

This is more than privacy. Privacy says: do not expose me. The model-of-me interface says: do not misrecognize me without recourse.

No one should be governed by a profile they cannot read.

This will be technically annoying.

Good.

Some human rights are technically annoying. That is how we know they are doing work.

A model inspection interface should not dump feature vectors on citizens like a truck unloading gravel. “Here are your embeddings, enjoy the dignity.” That is transparency as punishment. The interface must translate the model into meaningful claims.

The system thinks I prefer remote contribution.

The system thinks I am at risk of burnout.

The system thinks my writing project has low immediate civic return.

The system thinks I am unlikely to attend evening meetings.

The system thinks I respond well to structured prompts.

The system thinks I speak for a community I do not speak for.

That last one matters.

Models do not merely summarize. They assign position.

A person must be able to say:

No.

Wrong.

Old.

Incomplete.

Context missing.

Do not use this for that.

Ask me before applying this elsewhere.

That correction should not vanish into a ticketing abyss where truth grows old waiting for triage.

It should change the model, or mark the disagreement, or trigger review when stakes are high.

Self-model inspection is not a feature.

It is a dignity interface.

The Right to Outgrow the Model

A model that cannot forgive becomes a prison with analytics.

People change.

This is inconvenient for systems and essential for life.

A teenager becomes an adult. A patient recovers. A worker retrains. A dissenter moderates, or radicalizes, or becomes a gardener, which is often the wisest form of politics. A person once flagged as unreliable becomes steady. A person once steady collapses under grief. A person who spent ten years in one profession becomes useful elsewhere. A writer pauses coding. A coder becomes a caregiver. A caregiver becomes an advocate. An advocate becomes tired and wants silence.

Systems like continuity.

Life insists on revision.

The builder must design for outgrowing.

Old inferences should decay.

Labels should expire.

Crisis data should be quarantined from ordinary identity.

Childhood records should not follow adulthood without cause.

Refusal should not become permanent suspicion.

A wrong recommendation should not haunt the next ten.

The system should distinguish history from present state, and present state from destiny.

This is forgiveness as architecture.

Not sentimental forgiveness. Not erasing harm. Not pretending consequences do not matter.

Forgiveness as the right not to be permanently governed by an old model.

The system must allow a person to become someone else.

That may be one of the most radical design requirements of the adaptive age. More radical than decentralization, more difficult than explainability, more threatening to institutions than privacy. Privacy limits what a system sees. Forgiveness limits how long the system gets to believe itself.

The citizen who has changed must not spend life begging old data for parole.

Explanation, Stakes, and Recourse

Builders love to discuss the tradeoff between explainability and performance.

This tradeoff is real.

It is also frequently used as a velvet curtain behind which accountability takes a nap.

Not every system needs the same explanation. The stakes determine the duty.

If the system recommends a new walking route, a light explanation may suffice. If it denies a benefit, flags risk, recommends medical escalation, deprioritizes a life path, or shapes access to work, care, housing, credit, education, or civic participation, then “the model says so” is not an explanation. It is an oracle costume.

The question is:

What kind of explanation does the person need?

Technical explanation?

Moral explanation?

Procedural explanation?

Evidence?

Alternatives?

Uncertainty?

Recourse?

Sometimes the citizen does not need to know every internal detail. He needs to know why the decision happened, what data mattered, what assumptions were made, how uncertain the system is, who is responsible, and how to challenge it.

A powerful opaque model may be acceptable as a warning signal in some contexts.

It may not be acceptable as final authority.

The more a system affects life chances, the less acceptable it is for its reasoning to vanish into performance.

This is not anti-performance.

Performance matters.

A medical model that performs poorly but explains beautifully is not ethical. It is a poet with a malpractice problem. But a high-performing model that cannot be contested can become a polished instrument of domination.

The design answer is layered:

Use powerful models where useful.

Constrain their authority.

Demand explanation proportional to stakes.

Require human-readable reasons for consequential decisions.

Separate recommendation from final judgment.

Track errors.

Let people appeal.

Publish limits.

Disclose uncertainty.

A system that cannot explain itself at the level required by its power should not have that power.

This sounds severe because people have been harmed by systems that sounded reasonable.

When Not to Automate

The most important automation decision is sometimes no.

Builders hate this.

No does not demo well.

No does not produce a roadmap.

No does not look impressive in procurement.

No rarely raises money, unless packaged as “strategic non-automation,” at which point civilization deserves whatever invoice it receives.

But not every human friction is a design flaw.

Some frictions are the remaining places where conscience enters.

A judge pausing before sentence.

A clinician sitting with uncertainty.

A teacher noticing that a child’s wrong answer is more interesting than the correct one.

A mourner taking too long.

A community arguing because the issue cannot be reduced without violence.

A citizen refusing the recommended path because meaning does not always travel through the efficient route.

Automation should be forbidden, constrained, or advisory in certain zones.

Final judgment in justice.

Irreversible medical decisions.

Child welfare removal.

Forced risk classifications.

Political persuasion.

Moral worth.

Grief, ritual, and end-of-life care.

Identity assignment.

High-stakes denial without human review.

This does not mean tools have no role. They may gather evidence, reveal patterns, check consistency, surface precedents, simulate outcomes, or warn of bias.

But final authority must remain answerable to human judgment and public reason.

When not to automate is not a Luddite question.

It is a boundary question.

A civilization that automates every difficult moral pause has not become intelligent.

It has become impatient.

Dark Patterns as Civic Poison

In consumer software, dark patterns are manipulations of interface: hidden opt-outs, confusing cancellations, guilt prompts, bait-and-switch buttons, privacy settings buried under decorative fog.

In civic systems, dark patterns become constitutional harm.

Making refusal hard.

Hiding opt-outs.

Nudging toward data sharing.

Framing surveillance as safety.

Defaulting people into life pathways.

Using emotional manipulation.

Designing shame into assistance.

Making appeal exhausting.

Making “later” impossible to find.

Making the most invasive option the path of least resistance.

Making the non-recommended path look irresponsible.

These are not mere UX sins.

They shape freedom.

In civic systems, a dark pattern is a quiet constitutional offense.

That line should be printed on the wall of every lab building public-interest technology, preferably above the coffee machine, where moral principles stand a chance of being seen before the third cup.

A dark pattern in commerce steals attention or money.

A dark pattern in adaptive civic infrastructure steals agency.

The person may technically consent. Technically refuse. Technically appeal. Technically switch. Technically inspect. Technically correct. But if every path of agency is made exhausting, while surrender is made smooth, the system has chosen domination and hired design to hide the fingerprints.

The builder’s obligation is to make agency usable.

Refusal should be easy enough to be real.

Appeal should be visible enough to be trusted.

Consent should be understandable enough to matter.

Alternatives should be presented without shame.

The user should not need the stamina of a constitutional lawyer to remain free.

Who Governs the Builders?

A system that governs citizens must itself be governed.

This should be obvious.

It is not.

Builders are very good at placing oversight outside the core flow. A review board here. A compliance report there. A values statement. A model card. A safety review. A public consultation whose main function is to convert dissent into a PDF no one reads after launch.

Real governance must touch the architecture.

Who approves high-stakes model changes?

Who audits recommendation effects?

Who sees refusal-rate data?

Who tracks whether certain groups are nudged harder than others?

Who can freeze a harmful rollout?

Who investigates incidents?

Who publishes appeal statistics?

Who discloses conflicts of interest?

Who protects whistleblowers?

Who represents affected citizens?

Who reviews the review board when it becomes fond of itself?

Possible mechanisms are not mysterious.

Independent audits.

Public-interest review boards.

Citizen panels.

Red teams.

System cards.

Data provenance logs.

Incident reporting.

Procurement transparency.

Open standards where possible.

Conflict-of-interest disclosure.

Whistleblower protection.

Appeal metrics.

External review.

But mechanisms only matter if they have teeth, independence, and enough budget not to become ethics theater. Ethics theater is the performance of accountability in which everyone wears the costume and the system exits unchanged.

The builders must be governed not because builders are bad.

They must be governed because builders are human.

They get tired.

Promoted.

Captured.

Rushed.

Flattered.

Afraid.

Certain.

They inherit incentives like everyone else. A developer under pressure to improve adoption may hide friction. A product lead under pressure to show outcomes may overweight measurable returns. A public agency under pressure to reduce costs may call rationing “optimization.” A vendor under pressure to renew contracts may discover that dependence is a feature.

Good people cannot be the security model.

Governance is how good people remain good enough after the deadline arrives.

The Dangerous Word: Optimization

Optimization is where morality goes to become math and then pretends it has disappeared.

This does not mean optimization is bad. Optimization gave us efficient logistics, better energy use, improved medical scheduling, safer routes, lower waste, and all sorts of practical miracles. Anyone who has waited six hours for a badly scheduled appointment should not sneer at optimization too cheaply. Disorder is not humanism.

But optimize what?

Engagement?

Health?

Safety?

Productivity?

Compliance?

Cost?

Social contribution?

Happiness?

Carbon?

Institutional convenience?

Well-being?

Every objective function is a moral compression.

Something is chosen as target. Something else becomes constraint. Something else becomes noise. Something else disappears entirely because it resists measurement or refuses to file its meaning in the proper format.

The adaptive civic assistant in the review room optimized for social return, sustainability, contribution, health, and need. Reasonable goals. Good goals, even.

And yet, there was the phrase:

low-return creative work

The danger was not that the system hated creativity. The danger was that creativity had entered the model under a category unable to hold it.

Who defined the objective?

Who can contest it?

What gets sacrificed?

Does the person share that objective?

Is the objective local or imposed?

Can multiple objectives coexist?

What is the cost of being optimized for something you do not believe is the purpose of your life?

The builder must make objectives visible and revisable.

Not every value conflict can be solved by user preference sliders, though the industry will certainly try, and will likely call it something like “Personal Meaning Calibration.” But the system can at least disclose what it is optimizing, allow counterweights, preserve non-optimized paths, and prevent one metric from becoming sovereign.

Optimization without plural values becomes a quiet empire.

Exit, Portability, and Plural Help

A helpful system becomes tyrannical faster when there is no exit.

Exit is not the whole of justice. The ability to leave a bad system does not excuse the system’s harm, especially when leaving is costly, confusing, or available only to the privileged. “You can always leave” has been the lullaby of many landlords, employers, platforms, empires, and bad marriages.

But without exit, help curdles into ownership.

A citizen should be able to switch assistants.

Export relevant data.

Carry history without carrying every inference.

Move identity credentials without surrendering all context.

Transfer delegated functions.

Choose a public-interest fallback.

Use local alternatives.

Keep access to essential services even when rejecting optional personalization.

Leave the system without becoming illegible to society.

Portability must be designed before dependence forms. If people become dependent and only then discover exit is costly, portability has failed. Data portability, model portability where appropriate, interoperable identity, open civic standards, and plural providers are not technical niceties. They are anti-capture infrastructure.

No monopoly life-management layer.

That should be a constitutional sentence of the adaptive age.

There may be many helpers.

Public.

Community-based.

Private.

Personal.

Clinical.

Educational.

Civic.

Religious.

Family-centered.

Machine-assisted.

Human-only.

A humane future does not require one assistant to rule them all, one profile to bind them, one interface to gently guide them into optimized compliance. It requires plural help under shared rights floors.

Exit is not enough.

But without exit, the door becomes painting.

The Developer’s Oath

By the end of the review, the team had not solved the system.

This was encouraging.

A solved system in one afternoon is either trivial, fictional, or about to harm someone at scale.

What changed was smaller and more important.

The cooperation reliability score was removed from refusal behavior.

The “low-return creative work” phrase was marked for redesign, then ceremonially mocked, which is one of the healthier forms of institutional learning.

Refusal paths were promoted.

Self-model inspection was moved from future enhancement to launch requirement.

Delegation scopes were narrowed.

Consent expiry was added.

A high-stakes recommendation review layer was proposed, then sent to a working group, where good ideas go to either mature or die of meeting exposure.

On the whiteboard, someone wrote a draft oath for developers of adaptive civic systems.

It was not poetry.

Good.

Poetry near code should be used carefully, like soldering irons and national myths.

The oath read:

Do not infer what you do not need.

Do not hide what you infer.

Do not punish refusal.

Do not make help irreversible.

Do not optimize away personhood.

Do not design for saints.

Do not let convenience become coercion.

Do not let the model finish the person.

I would add one more:

Do not build a system that depends on your goodness in order for others to remain free.

Everything else is implementation detail.

Handoff: The Ledger Beneath the Help

The citizen in Chapter 24 asked for contestability.

The developer must build it.

But once the system can be inspected, refused, corrected, and escaped, the next question arrives immediately.

What does the system count as value?

Because a helpful system that preserves agency but runs on extractive value will eventually route people back into the old machine with better manners. It will protect refusal while rewarding hoarding. It will explain itself while optimizing scarcity. It will honor plural identity while measuring worth through old proxies. It will let the citizen contest the interface while the ledger underneath continues counting apples in locked closets until rot files for incorporation.

The developer can keep the system from becoming a tyrant.

But the next tyrant may be quieter.

The ledger that tells the system what matters.

Chapter 25 begins there.