A static profile says what a user has liked, searched, viewed or bought. A sequence says how that interest evolved, and how recently. After a LinkedIn feed-ranking conversation, I ran a small synthetic PoC to test whether order and elapsed time actually add predictive signal.

Context: In August 2026, Software Engineering Daily hosted LinkedIn VP of Engineering Tim Jurka on How LLMs Are Reshaping Recommendation Systems. The useful idea was not “replace the ranker with an LLM”. It was: treat the user’s path as a sequence, closer to next-token prediction than to a bag of interests. LinkedIn’s Feed Sequential Recommender (Feed SR) paper reports a 2.10% increase in time spent in online A/B tests after replacing a DCNv2 ranker. I did not want to copy that architecture. I wanted a smaller, falsifiable question I could answer with code.

Does preserving sequence and elapsed time improve next-topic prediction over simple popularity and static-preference baselines?

This article is that experiment. Synthetic data, chronological hold-out, five models, one shuffle control. Personal lab - not a production ranker.

The experiment in one picture

Recommendation does not have to be framed only as which item is most relevant to this user? It can also be framed as given the path this user has taken so far, what is most likely to be useful next?

That sounds like a small change in wording. It is not. Order changes meaning.

The PoC deliberately avoids Transformers at first. It compares five progressively more informed models:

1. Global popularity - what is popular across everyone?
2. Personal frequency - what has this user historically consumed most?
3. Ordered transition - given the latest topic, what usually comes next?
4. Gap-aware sequential - given the latest topic and the elapsed time, what usually comes next?
5. Shuffled-order control - what happens if I intentionally destroy chronology?

The fifth model is the real test. If the sequential model performs equally well after I shuffle each user’s history, I have not proved that sequence matters.

What the data looks like

The dataset is synthetic by design. I generated 1,000 users and 26,020 timestamped interactions using a fixed seed, making repeated runs deterministic in my local implementation. Exact reproduction would also require publishing the generator and evaluation code.

Each event contains four fields: user_id, timestamp, topic, action.

Here is a sample journey for one generated user:

A static model sees a collection of topics and frequencies. A sequence-aware model sees something closer to a journey. The March 11 Python event also shows why time may matter: a transition after one day may mean “continue the current task”; a transition after three weeks may mean “the user has returned with a new goal.”

Why synthetic data? I do not have a clean, large, publishable history of my own reading. Pretending that I do would make the story sound more personal and the experiment less honest. Synthetic data is appropriate for demonstrating the mechanism, evaluation design, and failure modes. It is not evidence that real users will produce the same gains.

How the journeys were generated

The generator contains several plausible technical learning paths. One runs Cloud Fundamentals -> Containers -> Kubernetes -> Platform Engineering -> Internal Developer Platforms -> Developer Experience -> FinOps. Another runs Python -> Data Engineering -> Embeddings -> Recommender Systems -> Sequential Recommendation -> LLM Ranking -> Evaluation.

Users do not move perfectly from left to right. The generator allows repetition, revisiting earlier concepts, skipping ahead, exploration, and longer gaps. A perfectly deterministic sequence would make the experiment trivial.

Evaluation: hide the future

For every user, interactions are sorted chronologically. The last event is held out as the test target. If the history is A -> B -> C -> D -> E, the model receives A -> B -> C -> D and must rank E as highly as possible. For the gap-aware model, the held-out event’s timestamp represents the simulated recommendation-request time; the held-out topic and action remain hidden.

I use Hit@3 as the primary metric: is the true next topic in the model’s top three predictions? I also report MRR@3 and NDCG@3. This is a per-user chronological split, not a global temporal cutoff. It prevents future events from the same user entering that user’s training history, but it does not model catalogue-wide changes over calendar time.

Five models

1. Global popularity recommends the same most-popular topics to everyone. Intentionally simple, and useful: a recommender that cannot beat popularity has not yet earned its complexity.

2. Personal frequency knows the user, but not the order of the user’s history. It asks what topics this person has interacted with most often. That is a traditional static preference profile.

3. Ordered transitions preserve chronology. For every adjacent pair of topics in the training histories, the model counts transitions and estimates P(next_topic = j | current_topic = i). A first-order Markov recommender:

for user in users:
    history = sort_by_time(user.events)
    for current, nxt in adjacent_pairs(history):
        transitions[current][nxt] += 1

At prediction time, the current topic becomes the context. No embeddings. No GPU. No attention. That simplicity is intentional.

4. Gap-aware sequential adds a deliberately crude time signal. Each transition belongs to one of two buckets: short gap (<= 7 days) or long gap (> 7 days). The model therefore learns two versions of its transition patterns: P(next | current, short_gap) and P(next | current, long_gap).

If I read about Kubernetes yesterday, platform engineering may be the natural continuation. If my last Kubernetes interaction was two months ago, the next topic may be driven more by a new goal than by the old session.

gap_days = (next_time - current_time).days
bucket = "short" if gap_days <= 7 else "long"
transitions[bucket][current][next_topic] += 1

This is not sophisticated temporal modeling. It is a probe: does time add any signal at all?

5. Shuffled-order control keeps the same topics but shuffles their order before learning transitions. The resulting model still knows which topics co-occur in similar users. What it loses is the actual journey. If performance barely changes, chronology was not doing much work.

Results

The deterministic run uses seed 42 and evaluates 1,000 held-out next events.

The absolute numbers are less important than the controlled differences.

Static preference -> sequence. Moving from personal frequency to ordered transitions increases Hit@3 from 0.670 to 0.794: +12.4 percentage points. The paired bootstrap 95% CI for the improvement is about +9.4 to +15.6 points. Inside this synthetic environment, knowing what tends to follow what adds substantial information beyond knowing what the user tends to like.

Sequence -> sequence + time. Adding the short/long gap feature increases Hit@3 from 0.794 to 0.819: another +2.5 percentage points (paired 95% CI about +1.2 to +3.8). That prevents an exaggerated conclusion. The experiment does not say that time is the dominant factor. It says:

Order carries most of the gain. A crude time signal adds a smaller incremental gain.

That is a more interesting result than forcing the article to support a “time-aware AI changes everything” headline.

The strongest evidence: shuffle the history

The gap-aware sequential model achieves Hit@3 = 0.819. The shuffled-order control achieves 0.655. The difference is +16.4 percentage points (95% CI: +13.5 to +19.3).

The shuffled model is not useless. Related topics still co-occur in the same users, so it retains some signal. But when chronology is destroyed, a large amount of predictive power disappears. That is the core finding of this PoC.

A concrete time-aware example

One generated user had this recent history:

Notice the 14-day gap before the held-out target.

Personal frequency and ordinary transitions both ranked FinOps, Cost Optimization, Autoscaling. The gap-aware model ranked FinOps, SLOs, Observability. It still did not rank the correct topic first, but it moved it into the top three.

That is the kind of behavior the time feature is intended to capture: after a longer break, the local transition pattern can differ from the immediate-session pattern. One example does not prove the model is generally better. The aggregate evaluation above is what matters. The example makes the mechanism easier to see.

Why not a Transformer first

It would have been easy to build a tiny SASRec-style model and make this article look more advanced. That would have weakened the experiment. Before adding a high-capacity architecture, I want to know whether simple signals already explain most of the result.

This question is especially relevant now. A 2026 Spotify Research paper, Do Sequential Recommendation Benchmarks Really Require Higher-Order Sequence Modelling?, evaluates strong recency-weighted pairwise methods against Transformer-based sequential recommenders. On several commonly used benchmarks, the simpler probes match or outperform reproduced Transformer baselines. MovieLens-20M is a clear exception, where the Transformer retains a substantial lead.

The lesson is not “Transformers are unnecessary.”

The lesson is:

A complex model should have to beat strong simple baselines that already capture popularity, recency, and pairwise transitions.

That is why this PoC begins with a Markov-style transition model rather than a neural network. If a future Transformer beats it materially on the same chronological evaluation, then we have evidence that higher-order context contributes something real.

What this PoC shows - and what it does not

What it demonstrates. Within the generated dataset: a static personal profile beats global popularity; preserving chronology beats the static profile; adding a simple elapsed-time feature improves the sequential model slightly; destroying chronology significantly reduces performance.

That means the setup isolates three different kinds of signal: preference, plus sequence, plus time.

What it does not prove. The dataset deliberately contains sequential structure. A sequence-aware model should discover it. The measured gains - 12.4 points, 16.4 points, or any other number - must not be presented as expected production uplift.

Real recommendation systems are harder: users have multiple simultaneous interests; items are dynamic and often new; actions have different intent strengths; candidate retrieval and ranking are separate problems; relevance is only one objective. Diversity, freshness, safety, creator quality, fairness, and long-term value also matter. Offline metrics do not guarantee online product gains.

The honest conclusion is narrower:

The PoC demonstrates a method for testing whether chronology and time carry predictive signal. The next step is to run the same ablation on real interaction logs.

From PoC to a real architecture

A production system would be much larger than this experiment, but the conceptual layers remain understandable:

At small scale, the “sequential ranker” could literally be the transition model used here. At large scale, that component might become a Transformer or another history encoder. The long-term profile, context features, policy rules, and product objectives can then be fused into the final ranking decision.

The architecture can become complex. The measurement principle should remain simple:

Does the new model beat strong static, recency, and transition baselines on chronological offline evaluation - and then survive an online experiment?

What I would build next

The next step should not be “add more layers.” I would proceed in this order:

1. Replace synthetic histories with real chronological events
2. Weight actions differently: view, then like, then save
3. Use several recent history items with recency decay
4. Add context such as device, session, or time of day if relevant
5. Build a SASRec-style or Transformer model
6. Compare it against the strongest simple sequential baseline
7. Only then consider production serving and online testing

The third step is particularly important. Instead of conditioning only on the latest topic, a stronger non-neural model can aggregate evidence from several recent interactions:

score(candidate) =
    w1 * P(candidate | last_item)
  + w2 * P(candidate | item_before_last)
  + w3 * P(candidate | older_item)

w1 > w2 > w3

That produces a much stronger test for whether a Transformer is learning genuine higher-order structure rather than simply reproducing recency-weighted pairwise statistics.

Closing

I started this experiment because of an LLM discussion. The useful lesson is not really about LLMs. It is about representation.

A user profile represented as {Kubernetes, AI, FinOps, Platform Engineering} is not equivalent to Kubernetes -> Platform Engineering -> Developer Experience -> FinOps. The first says what exists in the history. The second also says how the history evolved. When timestamps are included, we can begin to distinguish an active journey from an old preference.

The sentence I would keep from this entire PoC:

A user’s history is not only a list of preferences. It is an ordered path through changing intent.

The experiment is still small and synthetic. That is fine. A PoC should not pretend to be production evidence. What it gives me is something more useful than a copied architecture: a falsifiable method.

Preserve the sequence. Measure it. Destroy the sequence. Measure again. Add time. Measure again. Then increase model complexity only when the data justifies it.

That is the part I would carry into a real recommendation system.

References

• Software Engineering Daily, How LLMs Are Reshaping Recommendation Systems, 18 August 2026
• Hertel et al., An Industrial-Scale Sequential Recommender for LinkedIn Feed Ranking, arXiv:2602.12354, 2026
• Petrov et al., Do Sequential Recommendation Benchmarks Really Require Higher-Order Sequence Modelling?, arXiv:2608.19833, 2026
• Spotify Research, sequential-capacity-probes

PoC details: 1,000 synthetic users, 26,020 interactions, chronological leave-last-event-out evaluation, seed 42, Hit@3 / MRR@3 / NDCG@3, and bootstrap confidence intervals. The data and results are synthetic and should be interpreted as a demonstration of methodology rather than expected production uplift.

Pipelines still ran. Dashboards still loaded. Nobody deleted anything, fixed definitions, or admitted the mart nobody used cost thousands a month. That was a graveyard - not a platform.

Context: By mid-2026 many enterprises had years of cloud data investment and dozens of AI experiments - and the same failure mode kept showing up: no clear ownership for data products. I have seen this as much as a leadership problem as an architecture problem.

How graveyards form

Migration ends; SI leaves; runbooks thin; bus factor is one. Hero engineer moves teams; knowledge walks out. “Shared responsibility” - everyone assumes someone else owns freshness. New initiatives layer on: shadow marts, duplicate pipelines, more indexes. Cost grows quietly; nobody deletes because nobody will sign the deletion.

The platform looks alive. Accountability is dead.

Symptoms

Incidents start with “who owns this table?” Same metric built three ways with three numbers. Pipelines green while quality checks failed for weeks. Access requests go to a generic mailing list. Decommission dies in committee. AI teams rebuild customer 360 for every project.

Ownership is capacity to change, not blame

One person can wear multiple hats in a small team. Zero people wearing them is the graveyard pattern.

Contract fragment I pushed for on critical tables:

dataset: curated.orders
owners:
  business: order-management
  technical: data-platform-core
consumers:
  - bi_revenue_dashboard
  - support_copilot   # automated consumer = ownership risk
decommission_candidate: false
monthly_cost_eur_est: 4200

What I push for in the leadership seat

Name owners before publishing datasets. No owner -> not production for critical tiers.

Measure cost and usage per domain. Unused assets with cost are decommission candidates - with owner approval.

Treat decommissioning as a success metric. Reward turning things off after cutover.

On-call follows the data product for critical paths - same seriousness as customer-facing APIs.

When agents depend on a table, ownership includes that automated consumer risk. Put it in the contract.

Cleaning a graveyard without stopping the business

Month 1: Inventory - datasets, last access, monthly cost, named owner or NONE.

Month 2: Stabilize ownership on the top revenue/compliance products first.

Month 3: Merge duplicate marts; delete unused staging with sign-off; document lineage.

Ongoing: No new production without owner fields. Platform gates the deploy.

Unglamorous. Also what separates platforms that scale AI from platforms that accumulate debt.

Link to the technical leader trap

Senior engineers often become the implicit owner of everything. Feels helpful. Creates a graveyard in one person’s head.

Leadership means refusing to be the permanent owner, making ownership visible and rewarded, escalating unowned critical assets, building standards so ownership is lighter - not heavier. Solving everything yourself until you are the bottleneck is the other side of the same problem.

Closing

A platform without ownership becomes a graveyard: running pipelines, stale trust, rising cost, nobody accountable for change. Naming owners, measuring usage and cost, decommissioning on purpose, treating data products like production services - that is what helped.

The demo worked. The pilot dashboard looked green. Then production asked for SLAs, cost predictability, audit trails, and the same answer on Tuesday that you got on Monday.

Context: By spring 2026, the industry story was pilots to production. The graveyard I kept seeing was full of prototypes that never crossed the gap - not because the model was wrong, but because the operating system around the model was never built.

The prototype trap

Prototypes optimize for visible magic: hand-picked examples, clean indexes, tolerant users, no formal SLO or cost ceiling, manual cleanup when outputs look bad.

Production optimizes for reliability under variance: messy data, bad inputs, regulated and revenue-critical decisions, FinOps and audit, users who trust confident wrong answers.

Skills overlap partially. Infrastructure and discipline overlap less than teams expect.

Seven failure modes in the gap

1. No production definition of “good enough”. Pilot: “users liked it.” Production needs measured time saved, error/override rate, cost per task, p95 latency - with kill criteria. Without metrics you cannot tell if v2 is better.

2. Freshness and scope drift. Prototype index built once. Production tables and docs change hourly. Answers go wrong silently. Freshness SLAs, re-index pipelines, as_of in context.

3. Cost curves that do not scale. Demo: 50 queries/day. Production: 50,000 with retrieval, re-ranking, multi-step agents. Cost per successful outcome, budgets, caching, right-time tiers, route by complexity.

4. Missing fallbacks. Model or retrieval fails -> errors or hallucinations. Deterministic fallbacks, kill switches, degraded UX - tested, not improvised.

5. Security added late. Broad access “to move fast”. Agents compound exposure. Least privilege, retrieval boundaries, approval tiers.

6. No ownership after handoff. Data science built it; ops inherits it; data platform was never in the room. Named owners for model, inputs, runtime.

7. Change management ignored. Users do not know when to trust, when to override, what changed in v2. Release notes, confidence cues, feedback loops.

Gate I would use before go-live

Fail a critical row -> stay in bounded pilot. Do not call it production.

Pattern that worked

From personal lab copilot experiments and common enterprise patterns:

1. One narrow workflow with clear baseline metrics
2. Right-time freshness - not fake real-time
3. Strict success gates - retire what does not pass
4. Kill switch - prove reversibility
5. Platform fixes in parallel - not “after AI succeeds”

Production AI is systems integration. The model is one component.

After demo applause

Words I used:

The demo proved the workflow is worth testing. Production needs the same gates as customer-facing services: data ownership, cost ceiling, fallback, audit. Here is the 8-week path and what we will not do until gates pass.

Red flags: warehouse-wide access for the agent; no business owner for definitions; success metric is “delight” with no baseline; platform team learns about the project from security review.

What helped organizationally: single intake with risk tiers; platform + product + security sign-off for production; explicit retire list so failed pilots stop and budget returns to the platform.

Leaders who only celebrate demos inherit incidents. Leaders who gate honestly build durable capability.

Closing

AI projects fail between prototype and production when teams ship demos into operational reality without SLOs, cost models, fallbacks, governance, and ownership. Explicit production gates closed that gap - not another hackathon.

Everyone is talking about AI. The real question is whether your team builds systems that last - or relies on a few heroes, knowledge nobody wrote down, and demos that fall apart in production.

Context: By spring 2026, boards were asking engineering leaders about headcount, coding assistants, and whether they were “AI-native”. Engineers worried about staying relevant. Both miss the real point: AI makes strong engineering stronger, and weak engineering easier to see.

What AI was actually good at in engineering orgs

Boilerplate and scaffolding. Explaining unfamiliar code. Drafting tests, docs, SQL, Terraform. Faster exploration and PoCs. Summarizing incidents and logs.

That compresses local task time - especially for people who already know what “good” looks like.

What it did not replace

Accountability for production outcomes. Trade-off judgment under business constraints. Cross-team negotiation - contracts, priorities, decommissioning. Trust repair after an incident. Knowing what not to build. Operating messy legacy while migrating safely.

Orgs pay for judgment under uncertainty, not token output.

Weak engineering it exposed

AI did not create these problems. It raised the price of ignoring them.

Quick “is this platform ready for a pilot?” score I used in conversations. Tick the row if it is true today:

Score under 4 -> lab only. 4–5 -> bounded pilot. 6 -> discuss a production gate. Do not skip the empty rows because a demo looked good.

Strong engineering it amplified

Serving layers and contracts - retrieval and features cheaper to consume. Runbooks and tests. Platform standards so suggestions match house patterns. Measurement culture - eval harnesses and cost per task. Distributed ownership - fewer “only Maria knows” bottlenecks.

Mature platforms turn AI into leverage. Immature ones turn it into noise at scale.

What I saw leaders change their mind on

Stop asking only: how many engineers can we cut with AI?

Start asking: where does tacit knowledge block delivery? Which platform gaps make every AI project bespoke? What production gates do pilots skip? How do we measure value (time saved, errors, cost) - not demo applause? Which workflows should not be automated yet?

Headcount planning without platform maturity is guesswork.

What I tell individual contributors

I would not compete with the model on typing speed. What mattered more: system design and failure modes, data and domain understanding, production operation, clear writing and decision records, mentoring and review quality.

The engineer who only typed fast was already replaceable - by offshore, contractors, or templates. AI changes the margin, not the fundamentals.

Where I would still invest

Cutting platform investment while buying AI tools is the expensive path.

Closing

AI will not replace engineers who build accountable, operable systems. It will expose teams that relied on heroics, ambiguous data, and demo culture. Treat it as a stress test of engineering maturity - and fix the platform and ownership problems it reveals.

The promotion did not come with a new toolchain. It came with ambiguity, stakeholders who disagree, and problems that do not fit in an IDE.

Context: In 2026 many orgs created staff engineer and technical lead paths without clear definitions. When I moved into those roles the question was always: what actually changes on Monday? Notes from platform, data, and cloud leadership - not generic management theory.

What does not change

You still need technical judgment. You still read code, query plans, incident timelines. Credibility comes from understanding the work. If you stop being technically credible, good engineers route around you.

What actually changed for me

Calendar became the product. Experts optimize focus for deep work. Leaders optimize throughput: who is blocked, which decision is stuck, which risk is invisible upstairs, which standard would kill ten Slack threads. If 80% of the calendar is meetings, they need to produce owners, dates, and written decisions - not only “alignment”.

Communication is a deliverable. Expert: “Partition pruning fails on this query.” Leader: “Finance dashboard may breach SLA; owner is data-platform; fix or workaround by Thursday; impact if numbers are wrong.” Same fact. Different accountability.

Team graph, not call graph. Success is no longer “I shipped the fix”. It is: runbook used without you; junior owns a production change with review; two teams stop arguing because a contract exists; incident repeat rate drops.

Durable over perfect. Can we operate this at 3am? Can another team extend it without calling us? Can we reverse it? Does ownership survive my next vacation?

Translate both ways. Upward: risk into cost, revenue, compliance, timeline. Downward: business pressure into prioritized constraints, not vague urgency. Failure mode is absorbing anxiety from both sides until you burn out. Pass clarity, not stress.

Skills I had to practice on purpose

Written decisions (one-page ADRs). Facilitation with a named decider. Delegation with owner + checkpoint + authority. Hiring and mentoring so someone else can take your old tasks. Saying no with trade-offs written down. Enough financial literacy to talk cost-to-serve and ROI of platform work.

No MBA required. Reps required.

Mistakes in the first year

Still competing with the team on coding speed. Avoiding ownership and priority fights. Over-engineering standards nobody reads - one useful runbook beats a 40-page manifesto. Under-communicating to executives (surprises destroy trust faster than bad news told early). Identity crisis: “Am I still an engineer?” Yes - with a broader definition of impact.

Weekly rhythm that helped

Questions for your manager

What outcomes define success for me in six months? Which decisions are mine vs shared? What should I stop doing that I did as an IC? How will you measure team health, not only dates?

If they cannot answer, negotiate clarity before accepting infinite responsibility.

Closing

Expert -> technical leader means primary output becomes clarity, ownership, standards, and leverage - not personal execution speed. Still engineering. The interface is human and organizational as much as technical.

Governance decks showed up after the first incident. By then the architecture had already decided what agents could read, what was logged, and what could not be undone.

Context: Early 2026 AI governance stopped feeling theoretical in organizations I watched. Audit pressure and production agents pushed the same conclusion: controls have to live in systems, not only in policy PDFs.

The gap I kept seeing

Most orgs had a policy committee, a principles page, a review for “high-risk” use cases. Fewer had enforced access on retrieval indexes, immutable audit for automated decisions, model and data version pins, or controls that actually block a deploy when violated.

Governance you cannot verify in code and logs is theatre.

Controls that worked when they were in the architecture

Classification drives behavior. Tag datasets and documents (public_internal, confidential, regulated, pii). Pipelines and agents read those tags for indexes, models, human review, retention. A spreadsheet does not count.

Lineage for every production output. Model id + version, prompt/template version, source ids + versions, as_of, user or service identity. Store in queryable tables so audit can reconstruct in minutes - not only scattered logs.

-- illustrative lineage / inference log (personal lab style)
CREATE TABLE IF NOT EXISTS ai.inference_events (
  event_id STRING,
  ts TIMESTAMP,
  workflow STRING,
  model_id STRING,
  model_version STRING,
  prompt_template_version STRING,
  source_dataset_ids ARRAY<STRING>,
  as_of TIMESTAMP,
  user_or_sa STRING,
  outcome STRING,
  cost_eur NUMERIC
)
PARTITION BY DATE(ts)
CLUSTER BY workflow, outcome;

Access as code. IAM, row filters, masks, retrieval scopes in version-controlled config. Avoid: “the agent uses a service account that can read most of the warehouse.”

Human-in-the-loop by risk tier. Read-only with logging. Draft/suggest with sampling review. Acts on systems of record need an approval gate. Forbidden actions blocked at the tool layer. Risk tier is a property of the workflow, not a meeting outcome.

Eval and drift after launch. Golden-set regression on model/prompt changes. Sampled production quality. Alert when override rate or policy blocks spike. Kill switch drill on a schedule.

Questions auditors ask - answers architecture should already have

Engineers do not replace legal. They make compliance operable.

Anti-patterns

Governance review after production launch. One global “AI platform admin”. Logging prompts but not retrieved content hashes. Trusting a vendor “enterprise AI” badge without your own controls. Separate AI team with no data platform accountability.

A 30-day sequence I saw work

Week 1: classify datasets used by live or planned AI workflows.
Week 2: lineage fields on inference and retrieval logs.
Week 3: risk tiers + approval gate for one high-risk workflow.
Week 4: kill-switch drill + audit reconstruction exercise.

Small verifiable steps beat a twelve-month policy program.

Closing

AI governance belongs in architecture: classification, lineage, access, approval tiers, monitoring, kill switches. Principles on slides are necessary. Enforced system behavior is what keeps you out of the incident report.

When I was doing my PhD, I spent years building computational models for cancer detection. It sounded glamorous from the outside - algorithms, prediction, science. In reality it was a constant confrontation with risk and of course data as bigggg considration.

The data I worked with was deeply personal: gene expression profiles, clinical records, diagnostic outcomes. If my models were wrong, real people paid the price. A “good enough” model was not good enough. We had to think about:

• How the data was collected
• How it was stored and anonymised
• How predictions could be misinterpreted
• What could happen if something leaked

This mindset is exactly what I miss today when I look at how many companies are “doing AI”.

Right now, the narrative is simple and brutal:

Move fast, plug in AI everywhere, and worry about the details later.

From my perspective, that’s not innovation. That’s negligence dressed up as progress.

Table of Contents

1. Background: The Data Leakage Crisis
2. The Problem: Greedy AI Adoption
3. How Data Really Leaks Through AI
4. Methodology: A Risk-Based Framework
5. Implementation: Five-Step Secure AI Architecture
6. Technical Controls and Architecture
7. Governance and Policy Framework
8. Case Studies and Real-World Examples
9. Conclusion: AI with Discipline

Background: The Data Leakage Crisis

The Everyday AI That Quietly Eats Your Data

Let’s be honest about what’s actually happening inside every companies.

Employees are under pressure to deliver more, faster. They discover a large language model that can summarise, generate, and debug in seconds ( and honestly not eveyone underestand right know what are you doing???? ). So they start doing what feels natural:

• Paste internal documents into chatbots to “summarise this for me”
• Paste source code and logs to “help me debug this”
• Paste HR policies, contracts, even customer data to “make this clearer”

Nobody is trying to be malicious. People are just trying to survive their workday.

The problem is simple:

The moment that data leaves your controlled environment and lands in a third-party AI tool, you’ve lost control.

You don’t really know:

• How long the data is stored
• How big the data would stored
• what data is going to be stored
• Who exactly can access it
• Whether it might be used for training
• How it might resurface in another context
• ….

In my PhD world, that level of uncertainty would have killed any project immediately. In today’s corporate world, it often doesn’t even stop a pilot.

The Scale of the Problem

Recent studies and incidents reveal the scope:

• 2023: Samsung engineers leaked proprietary code via ChatGPT
• 2024: Multiple healthcare organizations exposed patient data through AI tools
• Ongoing: Thousands of companies have no policy on AI data usage

The pattern is consistent: well-intentioned employees, no guardrails, predictable outcomes.

The Problem: Greedy AI Adoption vs. Responsible Use

We can talk about ethics and principles, but let’s be direct: a lot of AI adoption is driven by greed and fear.

The Drivers

• Greed for valuation - “We are now an AI company” is cheaper than actually fixing your product
• Greed for KPIs - More users, more engagement, lower cost per ticket
• Fear of missing out - “If we don’t do it, our competitors will”

In this context, “AI strategy” often means:

1. Add AI to the landing page
2. Integrate with the first big LLM API available
3. Hope the security team signs off later

This might “work” short term, especially in startups desperate for growth. But structurally it creates a fragile system:

• No data classification
• No clear policy on PII
• No guardrails against employees leaking sensitive information
• No real incident response plan if something goes wrong

You end up with a strange paradox:

Companies shout about “AI transformation” while quietly gambling with the data that keeps them alive.

Human Safety and Company Safety Are Not Opposites

There’s a dangerous myth that human safety and business interests are in tension.

From what I’ve seen on both sides - research and industry - the opposite is true:

• If you misuse personal data, you lose customer trust
• If your AI behaves unpredictably in critical domains (health, finance, hiring), regulators and courts will eventually come
• If your internal knowledge, code, or strategy leaks through AI tools, your competitive advantage evaporates

Protecting people and protecting the company are the same problem with different time horizons.

In my PhD project, we could not hide behind “we are just experimenting”. We had to assume:

• Someone might act on these predictions.
• Someone might be harmed by them.
• Someone will ask us to defend our methods.

Companies deploying AI at scale should assume exactly the same.

How Data Really Leaks Through AI

Most leaks don’t look like a Hollywood hack. They look like normal work.

The Four Main Leakage Vectors

1. Accidental Pasting

Pattern:

• A developer pastes proprietary code into a chatbot
• A manager pastes an internal strategy deck for “summarisation”
• A recruiter pastes a list of candidates with full details to write “better outreach emails”

Why it happens:

• No clear policy on what’s allowed
• No technical controls preventing it
• Pressure to deliver quickly

2. Shadow Tools

Pattern:

• Browser extensions, plugins, or unofficial AI helpers that quietly send snippets of screen content to external services
• Personal accounts used for “quick help” on work documents

Why it happens:

• IT doesn’t control all tools employees use
• Personal devices and accounts blur boundaries
• No monitoring of data egress

3. Training Data Contamination

Pattern:

• Scraped datasets that include PII or confidential documents
• Models later regurgitate pieces of those documents when probed in specific ways

Why it happens:

• Training data sources not vetted
• No data minimization in model training
• Models retain more information than expected

4. Misconfiguration and Bugs

Pattern:

• Wrong access settings, test environments made public
• Shared logs that contain prompts and responses
• API keys exposed in code repositories

Why it happens:

• Rush to deploy
• No security review process
• Default settings are permissive

None of this is spectacular. It’s routine. That’s exactly why it’s so dangerous.

Data Leakage Risk Matrix

Methodology: A Risk-Based Framework

If a company genuinely wants to adopt AI without playing roulette with data and trust, it needs something more mature than “let’s try this API”.

Core Principles

1. Data Classification First - Know what you have before you protect it
2. Risk-Based Approach - Not all AI features are equal
3. Defense in Depth - Technical controls + policy + training
4. Measurable Outcomes - Track incidents, near misses, trust metrics
5. Reversibility - Kill switches and fallback mechanisms

The Framework Structure

Implementation: Five-Step Secure AI Architecture

A minimal, workable approach can be broken into five steps.

Step 1: Classify Your Data First

Before touching AI, be explicit:

Data Classification Levels:

• Public - Can safely be sent anywhere (marketing copy, public documentation)
• Internal Non-Sensitive - Inconvenient if leaked, but survivable (internal process docs, non-confidential policies)
• Confidential - Source code, architecture, contracts, internal strategy
• PII / Highly Sensitive - Customer data, HR data, health data, financial records

Clear Rules:

• Public: OK with external AI tools
• Internal Non-Sensitive: Only with approved tools and vendors
• Confidential & PII: Never to public AI; only via controlled, private models with contracts and technical safeguards

If a company can’t answer “what data do we actually have and how sensitive is it?”, then it has no business deploying AI on top of it.

Step 2: Classify Your AI Use Cases by Risk

Not all AI features are equal. A marketing text generator and an AI system deciding who gets a loan do not belong in the same risk category.

Risk Assessment Matrix:

Define two dimensions:

• Data Sensitivity (low / high)
• Human & Business Impact (low / high)

You then get four quadrants:

Decision Framework:

• Low sensitivity, low impact –> “Experiment freely, with basic hygiene”
• High sensitivity or high impact –> “Formal assessment, documented controls, human oversight, and very careful rollout”

Anything high-impact + high-sensitivity should be treated like a medical model: audited, tested, explainable, and reversible.

Step 3: Design AI Architecture With Controls, Not Just Features

A secure AI setup is more than “user –> LLM API”.

You need an internal layer that:

• Filters and redacts sensitive data where possible (data minimisation)
• Enforces policies (“no PII to external models”)
• Logs all requests and responses for audit and incident analysis
• Routes sensitive workloads to private / VPC-isolated models

Most companies go wrong by integrating AI at the edge (direct from browser to vendor). You want AI behind an internal gateway that you control.

Step 4: Set Policy, Ownership, and Training

People are not going to guess the right behaviour. You must be explicit.

Required Elements:

• A clear AI usage policy: what tools are allowed, forbidden, and under what constraints
• A clear owner (or committee) for AI risk and governance
• Regular training for employees: what is PII, what counts as confidential, concrete examples of “do” and “don’t”

If AI is “everyone’s job” but nobody is accountable, risk will accumulate silently until something breaks.

Step 5: Monitor, Learn, and Treat Incidents as Feedback

Assume mistakes. Then build for them.

Monitoring Requirements:

• Monitor AI usage and look for abnormal patterns
• Have a basic incident response plan: who is involved, what gets shut down, how you communicate
• After each incident or near miss, update policies and controls, not just the slide deck

Companies that pretend “nothing bad will happen” with AI are signalling that they have no realistic understanding of how technology fails in the real world.

Technical Controls and Architecture

Secure AI Data Flow Architecture

Implementation Example: AI Gateway Service

Architecture Components:

1. API Gateway - Routes all AI requests
2. Data Loss Prevention (DLP) - Scans and redacts sensitive data
3. Policy Engine - Enforces data classification rules
4. Audit Logger - Records all AI interactions
5. Routing Layer - Directs requests to appropriate AI provider

Example Implementation (Python/FastAPI):

"""
AI Gateway Service - Secure routing and policy enforcement
"""
from fastapi import FastAPI, HTTPException, Request
from pydantic import BaseModel
from typing import Optional, Dict, Any
import logging
from datetime import datetime

# Data classification
class DataClassification:
    PUBLIC = "public"
    INTERNAL = "internal"
    CONFIDENTIAL = "confidential"
    PII = "pii"

# DLP Service
class DLPService:
    """Data Loss Prevention - detects and redacts sensitive data"""
    
    def classify_data(self, content: str) -> str:
        """Classify data sensitivity level"""
        # PII detection patterns
        pii_patterns = [
            r'\b\d{3}-\d{2}-\d{4}\b',  # SSN
            r'\b\d{16}\b',  # Credit card
            r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',  # Email
        ]
        
        import re
        for pattern in pii_patterns:
            if re.search(pattern, content):
                return DataClassification.PII
        
        # Confidential keywords
        confidential_keywords = ['proprietary', 'confidential', 'internal strategy']
        if any(keyword in content.lower() for keyword in confidential_keywords):
            return DataClassification.CONFIDENTIAL
        
        return DataClassification.PUBLIC
    
    def redact_sensitive(self, content: str, classification: str) -> str:
        """Redact sensitive data based on classification"""
        if classification == DataClassification.PII:
            # Redact PII patterns
            import re
            content = re.sub(r'\b\d{3}-\d{2}-\d{4}\b', '[SSN-REDACTED]', content)
            content = re.sub(r'\b\d{16}\b', '[CARD-REDACTED]', content)
            content = re.sub(
                r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
                '[EMAIL-REDACTED]',
                content
            )
        return content

# Policy Engine
class PolicyEngine:
    """Enforces AI usage policies based on data classification"""
    
    def __init__(self):
        self.rules = {
            DataClassification.PUBLIC: {
                "allowed_providers": ["openai", "anthropic", "google"],
                "requires_approval": False,
            },
            DataClassification.INTERNAL: {
                "allowed_providers": ["openai-enterprise", "anthropic-enterprise"],
                "requires_approval": True,
            },
            DataClassification.CONFIDENTIAL: {
                "allowed_providers": ["private-vpc-model"],
                "requires_approval": True,
                "requires_audit": True,
            },
            DataClassification.PII: {
                "allowed_providers": ["private-vpc-model"],
                "requires_approval": True,
                "requires_audit": True,
                "requires_encryption": True,
            }
        }
    
    def check_policy(self, classification: str, provider: str) -> Dict[str, Any]:
        """Check if request complies with policy"""
        if classification not in self.rules:
            return {"allowed": False, "reason": "Unknown classification"}
        
        rule = self.rules[classification]
        
        if provider not in rule["allowed_providers"]:
            return {
                "allowed": False,
                "reason": f"Provider {provider} not allowed for {classification} data"
            }
        
        return {
            "allowed": True,
            "requires_approval": rule.get("requires_approval", False),
            "requires_audit": rule.get("requires_audit", False),
            "requires_encryption": rule.get("requires_encryption", False),
        }

# Audit Logger
class AuditLogger:
    """Logs all AI interactions for compliance and incident response"""
    
    def log_request(
        self,
        user_id: str,
        classification: str,
        provider: str,
        input_size: int,
        output_size: int,
        timestamp: datetime,
        policy_result: Dict[str, Any],
    ):
        """Log AI request to audit system"""
        log_entry = {
            "timestamp": timestamp.isoformat(),
            "user_id": user_id,
            "data_classification": classification,
            "provider": provider,
            "input_size_bytes": input_size,
            "output_size_bytes": output_size,
            "policy_allowed": policy_result["allowed"],
            "policy_reason": policy_result.get("reason"),
        }
        
        # In production: write to secure audit log (e.g., BigQuery, CloudWatch)
        logging.info(f"AI_AUDIT: {log_entry}")
        
        # Example: Write to database or SIEM
        # audit_db.insert(log_entry)

# Main Gateway Service
app = FastAPI(title="AI Gateway", version="1.0.0")

dlp_service = DLPService()
policy_engine = PolicyEngine()
audit_logger = AuditLogger()

class AIRequest(BaseModel):
    content: str
    provider: str = "openai"
    user_id: str

class AIResponse(BaseModel):
    result: str
    classification: str
    redacted: bool
    policy_compliant: bool

@app.post("/ai/generate", response_model=AIResponse)
async def generate_ai(request: AIRequest):
    """Secure AI generation endpoint"""
    
    # Step 1: Classify data
    classification = dlp_service.classify_data(request.content)
    
    # Step 2: Check policy
    policy_result = policy_engine.check_policy(classification, request.provider)
    
    if not policy_result["allowed"]:
        raise HTTPException(
            status_code=403,
            detail=f"Policy violation: {policy_result['reason']}"
        )
    
    # Step 3: Redact if needed
    redacted_content = request.content
    redacted = False
    if classification in [DataClassification.PII, DataClassification.CONFIDENTIAL]:
        redacted_content = dlp_service.redact_sensitive(request.content, classification)
        redacted = redacted_content != request.content
    
    # Step 4: Route to appropriate provider
    # In production: call actual AI provider API
    result = f"[AI Response for {classification} data]"
    
    # Step 5: Audit log
    audit_logger.log_request(
        user_id=request.user_id,
        classification=classification,
        provider=request.provider,
        input_size=len(request.content.encode()),
        output_size=len(result.encode()),
        timestamp=datetime.utcnow(),
        policy_result=policy_result,
    )
    
    return AIResponse(
        result=result,
        classification=classification,
        redacted=redacted,
        policy_compliant=True,
    )

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8080)

Data Classification Automation

Example: Automated Classification Service

"""
Automated data classification using ML and pattern matching
"""
import re
from typing import List, Tuple
from dataclasses import dataclass

@dataclass
class ClassificationResult:
    level: str
    confidence: float
    matched_patterns: List[str]
    recommendations: List[str]

class AutoClassifier:
    """Automatically classify data sensitivity"""
    
    def __init__(self):
        self.pii_patterns = {
            "ssn": r'\b\d{3}-\d{2}-\d{4}\b',
            "credit_card": r'\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b',
            "email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
            "phone": r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b',
            "ip_address": r'\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b',
        }
        
        self.confidential_keywords = [
            "proprietary", "confidential", "internal use only",
            "trade secret", "nda", "non-disclosure"
        ]
        
        self.code_indicators = [
            "def ", "function ", "class ", "import ", "package ",
            "SELECT", "INSERT", "CREATE TABLE"
        ]
    
    def classify(self, content: str) -> ClassificationResult:
        """Classify content and return result with confidence"""
        matched = []
        recommendations = []
        
        # Check for PII
        pii_found = False
        for name, pattern in self.pii_patterns.items():
            if re.search(pattern, content, re.IGNORECASE):
                matched.append(f"PII: {name}")
                pii_found = True
        
        if pii_found:
            return ClassificationResult(
                level=DataClassification.PII,
                confidence=0.95,
                matched_patterns=matched,
                recommendations=[
                    "Never send to public AI models",
                    "Use private VPC model only",
                    "Require encryption in transit and at rest"
                ]
            )
        
        # Check for confidential indicators
        confidential_found = any(
            keyword in content.lower() for keyword in self.confidential_keywords
        )
        
        # Check for code
        code_found = any(
            indicator in content for indicator in self.code_indicators
        )
        
        if confidential_found or code_found:
            return ClassificationResult(
                level=DataClassification.CONFIDENTIAL,
                confidence=0.85,
                matched_patterns=matched + (["confidential_keywords"] if confidential_found else []) + (["code_indicators"] if code_found else []),
                recommendations=[
                    "Use private model or approved enterprise provider",
                    "Require manager approval",
                    "Enable audit logging"
                ]
            )
        
        # Default to internal
        return ClassificationResult(
            level=DataClassification.INTERNAL,
            confidence=0.70,
            matched_patterns=matched,
            recommendations=[
                "Use approved enterprise AI providers",
                "Review output before use"
            ]
        )

Kill Switch Implementation

Example: Environment-Based Kill Switch

"""
Kill switch for disabling AI features instantly
"""
import os
from typing import Dict, Any, Callable, Optional
from enum import Enum

class KillSwitchStatus(Enum):
    ENABLED = "enabled"
    DISABLED = "disabled"
    DEGRADED = "degraded"  # Limited functionality

class KillSwitch:
    """Centralized kill switch for AI services"""
    
    def __init__(self):
        self.status = self._read_status()
        self.reason = os.getenv("AI_KILL_SWITCH_REASON", "")
    
    def _read_status(self) -> KillSwitchStatus:
        """Read kill switch status from environment"""
        status_str = os.getenv("AI_ENABLED", "true").lower()
        
        if status_str == "false" or status_str == "disabled":
            return KillSwitchStatus.DISABLED
        elif status_str == "degraded":
            return KillSwitchStatus.DEGRADED
        else:
            return KillSwitchStatus.ENABLED
    
    def is_enabled(self) -> bool:
        """Check if AI is enabled"""
        return self.status == KillSwitchStatus.ENABLED
    
    def get_fallback_response(self, service_type: str) -> Dict[str, Any]:
        """Get deterministic fallback when AI is disabled"""
        fallbacks = {
            "code": {
                "suggestions": [],
                "message": "AI code assistance is currently disabled. Please contact IT support.",
            },
            "document": {
                "summary": "",
                "message": "AI document processing is currently disabled.",
            },
            "chat": {
                "response": "AI chat is currently unavailable. Please try again later.",
            }
        }
        
        return fallbacks.get(service_type, {"message": "AI service unavailable"})
    
    def with_kill_switch(
        self,
        ai_function: Callable,
        service_type: str,
        *args,
        **kwargs
    ) -> Dict[str, Any]:
        """Execute AI function with kill switch protection"""
        if not self.is_enabled():
            return {
                "result": self.get_fallback_response(service_type),
                "ai_enabled": False,
                "status": "fallback",
                "reason": self.reason or "Kill switch activated",
            }
        
        try:
            result = ai_function(*args, **kwargs)
            return {
                "result": result,
                "ai_enabled": True,
                "status": "success",
            }
        except Exception as e:
            # On error, fall back gracefully
            return {
                "result": self.get_fallback_response(service_type),
                "ai_enabled": True,
                "status": "error",
                "error": str(e),
            }

Governance and Policy Framework

AI Risk Lifecycle

Policy Template: AI Usage Policy

Sample AI Usage Policy Structure:

AI Usage Policy

1. Scope

This policy applies to all employees, contractors, and third parties using AI tools for company business.

2. Data Classification

Public Data

• Definition: Marketing materials, public documentation, non-sensitive content
• Allowed: Public AI tools (ChatGPT, Claude, etc.)
• Requirements: Basic review of outputs

Internal Non-Sensitive

• Definition: Internal process docs, non-confidential policies
• Allowed: Approved enterprise AI providers only
• Requirements: Manager approval, audit logging

Confidential Data

• Definition: Source code, architecture, contracts, strategy
• Allowed: Private VPC models only
• Requirements: CISO approval, encryption, full audit trail

PII / Highly Sensitive

• Definition: Customer data, HR data, health data, financial records
• Allowed: Private VPC models with data processing agreements
• Requirements: CISO + Legal approval, encryption, anonymization, full audit

3. Prohibited Uses

• Never paste PII into public AI tools
• Never paste proprietary code into public AI tools
• Never use personal AI accounts for company data
• Never use unapproved browser extensions or plugins

4. Approved Tools

Public Tools (Public Data Only)

• ChatGPT (with company account)
• Claude (with company account)
• Google Gemini (with company account)

Enterprise Tools (Internal/Confidential)

• OpenAI Enterprise
• Anthropic Enterprise
• Microsoft Copilot (with data residency)

Private Tools (PII/Confidential)

• Private VPC-hosted models
• On-premise AI infrastructure

5. Incident Response

If you suspect data has been leaked:

1. Immediately stop using the AI tool
2. Report to security@company.com
3. Document what data was shared
4. Do not attempt to delete or cover up

6. Training Requirements

All employees must complete:

• Annual AI security training
• Data classification training
• Incident reporting procedures

7. Enforcement

Violations may result in:

• Disciplinary action
• Termination for willful violations
• Legal action for data breaches

Ownership and Accountability

RACI Matrix for AI Governance:

Legend:

• R = Responsible (does the work)
• A = Accountable (owns the outcome)
• C = Consulted (provides input)
• I = Informed (kept in the loop)

Case Studies and Real-World Examples

Case Study 1: Code Leakage Incident

Scenario: A software company’s engineers were using ChatGPT to debug code. Over several months, proprietary algorithms and architecture details were pasted into the tool.

Impact:

• Competitive advantage lost
• Potential IP violation
• Customer trust damaged

Root Cause:

• No policy on AI usage
• No technical controls
• No monitoring

Solution Implemented:

1. Immediate: Blocked public AI tools, required VPN for approved tools
2. Short-term: Implemented AI gateway with DLP
3. Long-term: Private code analysis tools, training program

Lessons:

• Code is always confidential
• Technical controls beat policy alone
• Monitor before you have an incident

Case Study 2: Healthcare PII Exposure

Scenario: A healthcare organization’s staff used AI tools to summarize patient notes. Patient names, conditions, and treatment plans were exposed.

Impact:

• HIPAA violation
• Regulatory fines
• Patient lawsuits

Root Cause:

• PII not properly classified
• No redaction before AI processing
• Staff not trained on data sensitivity

Solution Implemented:

1. Immediate: Shut down all AI tools, incident response
2. Short-term: Implemented PII detection and redaction
3. Long-term: Private healthcare AI model, comprehensive training

Lessons:

• Healthcare data requires special handling
• Automated redaction is essential
• Training must be role-specific

Case Study 3: Successful Implementation

Scenario: A financial services company wanted to use AI for customer support but had strict regulatory requirements.

Approach:

1. Classified all customer data as PII
2. Implemented private VPC-hosted model
3. Added full audit logging
4. Trained all support staff
5. Started with low-risk use cases

Results:

• Zero data leaks in 12 months
• 30% reduction in support ticket resolution time
• Regulatory approval maintained

Key Success Factors:

• Started with classification
• Technical controls from day one
• Continuous monitoring and improvement

What I Learned From Cancer Models That Applies to Corporate AI

Working on predictive models for cancer detection taught me a few non-negotiable rules:

1. The cost of a wrong decision matters more than the beauty of the model.
2. You can’t separate model performance from data governance.
3. You must be able to explain and defend your approach when it matters most.

I see too many companies obsess over model performance and UX while ignoring the basics:

• Where is the data coming from?
• Who controls it end-to-end?
• What happens if the worst-case scenario actually occurs?

When you’re dealing with people’s lives or livelihoods, “we were experimenting” is not an excuse. It’s a confession.

The Parallels

Conclusion: AI with Discipline

Three Clear Takeaways for Leaders and Teams

If you’re serious about AI and don’t want to be the next cautionary case study, treat these as the bare minimum:

1. Ban Confidential and PII Data from Public AI Tools - and Enforce It Technically

Policies without controls are theatre. You need:

• Automated data classification
• Technical controls (gateway, DLP, routing)
• Monitoring and alerting
• Incident response procedures

2. Use a Risk-Based Approach to AI Features

Don’t treat a marketing chatbot and an AI credit scorer as the same. They are not.

• Classify use cases by impact and sensitivity
• Apply appropriate controls for each quadrant
• Start with low-risk, high-value use cases
• Build capability before tackling high-risk scenarios

3. Measure Success with Safety and Trust Included

Track not only adoption and revenue, but also:

• Incidents and near misses
• Policy violations
• User trust metrics
• Cost of controls vs. cost of incidents

If those metrics are invisible, your AI programme is fundamentally incomplete.

The Choice

AI is not going away. The question is whether we force it to grow up - or we let greed and fear drive us into predictable disasters.

The choice is not “AI or safety”. The serious choice is:

AI with discipline, or AI with collateral damage.

Author: Dr. Atabak Kheirkhah
Date: November 30, 2025
Contact: atabakkheirkhah@gmail.com

Right-Time AI: Small Copilots, Big Proof - An End-to-End PoC

In a two-week personal side experiment, I built three micro-copilots (reading, meeting, and code) on Vertex AI, Cloud Run, Workflows, and BigQuery. I defined strict hypotheses (time saved, edits reduced, and predictable cost), mapped decisions to right-time freshness tiers (seconds, minutes, or daily), and added a kill switch for reversibility. Two copilots passed the stated thresholds (≥15% time saved and ≥20% fewer edits); one failed and was retired. This article shares the pattern, guardrails, and metrics to help teams decide what to keep, what to stop, and why.

Key Results:

• Reading Copilot: 17% time saved, 21% fewer edits, $0.01/task (daily batch)
• Meeting Copilot: 20% time saved, p95 latency <3 min
• Code Copilot: Mixed results - retired pending on-device experiments
• Kill Switch: 100% successful fallback rate
• Cost: Predictable $0.01-0.02 per task with no spikes

Table of Contents

1. Why I Did It
2. The Rules (Before Code)
3. Architecture
4. Implementation
5. Testing Strategy
6. Measurement & Metrics
7. Results
8. Governance & Trust
9. What Failed & Why
10. Reusable Pattern

Why I Did It

Usually demos are easy, but daily impact is hard. I wanted a personal, honest answer to a simple question: Can small AI helpers applied to real tasks save people time without creating chaos or surprise costs?

The goal wasn’t to build the next ChatGPT, Gemini, Copilot, or any other competitor. It was to test a repeatable, governed pattern that delivers measurable value-one that could be adapted to other personal or company workflows with clear success gates and a kill switch.

The Rules (Before Code)

Scope

• Three micro-copilots for workflows: reading, meetings, code
• No business data, no PII

Right-Time Freshness

• Code: seconds (interactive code review)
• Meeting: minutes (post-meeting processing)
• Reading: daily (overnight batch processing)

No fake “real-time” where decisions don’t need it.

Guardrails

• EU region only
• Plain-language prompts
• Audit logs on all operations
• One-click kill switch (AI_ENABLED=false)

Success Gates

• H1: ≥15% time saved on ≥2 copilots (median over 5+ runs)
• H2: Edits reduced by ≥20% vs. baseline
• H3: Stable cost/task with ceilings per use case
• H4: p95 latency meets SLA for its freshness tier
• H5: Reversibility proven (kill switch -> clean fallback, logged)

Architecture

System Overview

Component Details

I used a personal Gmail account and personal GCP trial credits for this test-not employer infrastructure or data. Details follow:

1. Cloud Run Services

Characteristics:

• Serverless, auto-scaling (0 to N instances)
• Request timeout: 300 seconds
• Container concurrency: 80
• CPU: 1 vCPU, Memory: 512Mi
• Cost optimization: Scale to zero, max 3 instances

Endpoints:

• POST /suggest - Main copilot endpoint
• GET /health - Health check with kill switch status

2. BigQuery Schema

Dataset: copilot_poc

Tables:

• events - One row per invocation with full telemetry
• catalog - Workflow registry with freshness tiers and SLAs
• metrics (view) - Daily aggregates by copilot

Partitioning: Events table partitioned by date

Clustering: By copilot and outcome for query optimization

3. Security & Governance

• IAM: Service accounts with least-privilege roles
• Secrets: Secret Manager for API keys
• Audit Logs: All operations logged
• Kill Switch: Environment variable AI_ENABLED=false
• EU Locations: Compute in europe-west4; analytics in BigQuery’s EU multi-region

Implementation

Shared Library: AI Client Abstraction

File: lib/ai_client.py

"""
AI client abstraction with a Vertex AI implementation.
"""
import os
import time
from abc import ABC, abstractmethod
from typing import Dict, List, Optional, Tuple
from enum import Enum

try:
    from google.cloud import aiplatform
    from vertexai.preview.generative_models import GenerativeModel
except ImportError:
    aiplatform = None
    GenerativeModel = None

class AIProvider(str, Enum):
    """Supported AI providers."""
    OPENAI = "openai"
    VERTEX = "vertex"

class AIClient(ABC):
    """Abstract base class for AI clients."""
    
    @abstractmethod
    def generate(
        self,
        prompt: str,
        system_prompt: Optional[str] = None,
        max_tokens: int = 2000,
        temperature: float = 0.7,
    ) -> Tuple[str, Dict[str, int]]:
        """
        Generate a response from the AI model.
        
        Returns:
            Tuple of (response_text, token_usage_dict)
            token_usage_dict contains: tokens_in, tokens_out
        """
        pass

class VertexAIClient(AIClient):
    """Google Vertex AI client."""
    
    def __init__(
        self,
        project_id: Optional[str] = None,
        location: str = "europe-west4",
        model: str = "gemini-2.5-flash",
    ):
        if GenerativeModel is None:
            raise ImportError(
                "vertexai package not installed. Install with: pip install google-cloud-aiplatform"
            )
        
        self.project_id = project_id or os.getenv("GCP_PROJECT_ID")
        if not self.project_id:
            raise ValueError("GCP_PROJECT_ID not provided")
        
        self.location = location
        self.model_name = model
        
        aiplatform.init(project=self.project_id, location=self.location)
        self.model = GenerativeModel(model)
    
    def generate(
        self,
        prompt: str,
        system_prompt: Optional[str] = None,
        max_tokens: int = 2000,
        temperature: float = 0.7,
    ) -> Tuple[str, Dict[str, int]]:
        """Generate response using Vertex AI."""
        full_prompt = prompt
        if system_prompt:
            full_prompt = f"{system_prompt}\n\n{prompt}"
        
        response = self.model.generate_content(
            full_prompt,
            generation_config={
                "max_output_tokens": max_tokens,
                "temperature": temperature,
            },
        )
        
        content = response.text
        usage = response.usage_metadata
        thinking_tokens = int(getattr(usage, "thoughts_token_count", 0) or 0)
        token_usage = {
            "tokens_in": int(usage.prompt_token_count),
            "tokens_out": int(usage.candidates_token_count) + thinking_tokens,
        }
        
        return content, token_usage

def get_ai_client(
    provider: Optional[str] = None,
    **kwargs
) -> AIClient:
    """
    Factory function to get the appropriate AI client.
    
    Args:
        provider: "vertex" (defaults to AI_PROVIDER env var)
        **kwargs: Additional arguments passed to the client constructor
    
    Returns:
        AIClient instance
    """
    provider = provider or os.getenv("AI_PROVIDER", AIProvider.VERTEX.value)
    
    if provider == AIProvider.OPENAI.value:
        raise NotImplementedError("OpenAIClient is not included in this PoC")
    elif provider == AIProvider.VERTEX.value:
        return VertexAIClient(**kwargs)
    else:
        raise ValueError(f"Unknown AI provider: {provider}")

Kill Switch Implementation

File: lib/kill_switch.py

"""
Kill switch logic for disabling AI and falling back to deterministic outputs.
"""
import os
from typing import Dict, Any, Callable, Optional

def is_ai_enabled() -> bool:
    """Check if AI is enabled via environment variable."""
    return os.getenv("AI_ENABLED", "true").lower() == "true"

def get_fallback_response(copilot_type: str) -> Dict[str, Any]:
    """
    Get deterministic fallback response when AI is disabled.
    
    Args:
        copilot_type: "code", "reading", or "meeting"
    
    Returns:
        Dictionary with fallback response structure
    """
    fallbacks = {
        "code": {
            "test_suggestions": [],
            "refactor_hints": [],
            "message": "AI is disabled. Please enable AI_ENABLED=true to get suggestions.",
        },
        "reading": {
            "digest": [],
            "follow_up_prompts": [],
            "message": "AI is disabled. Please enable AI_ENABLED=true to get digest.",
        },
        "meeting": {
            "action_items": [],
            "message": "AI is disabled. Please enable AI_ENABLED=true to get action items.",
        },
    }
    
    return fallbacks.get(copilot_type, {"message": "AI is disabled"})

def with_kill_switch(
    ai_function: Callable,
    copilot_type: str,
    *args,
    **kwargs
) -> Dict[str, Any]:
    """
    Execute AI function with kill switch fallback.
    
    Args:
        ai_function: Function to call if AI is enabled
        copilot_type: Type of copilot for fallback response
        *args, **kwargs: Arguments to pass to ai_function
    
    Returns:
        Result from ai_function or fallback response
    """
    if not is_ai_enabled():
        return {
            "result": get_fallback_response(copilot_type),
            "ai_enabled": False,
            "outcome": "fallback",
        }
    
    try:
        result = ai_function(*args, **kwargs)
        return {
            "result": result,
            "ai_enabled": True,
            "outcome": "success",
        }
    except Exception as e:
        return {
            "result": get_fallback_response(copilot_type),
            "ai_enabled": True,
            "outcome": "error",
            "error": str(e),
        }

Code Copilot Service

File: copilots/code_copilot/main.py

"""
Code Copilot FastAPI service.
"""
import os
import sys
from typing import Optional
from fastapi import FastAPI, HTTPException, UploadFile, File, Form
from fastapi.responses import JSONResponse
from pydantic import BaseModel

sys.path.insert(0, os.path.join(os.path.dirname(__file__), '../../'))

from lib.kill_switch import is_ai_enabled, with_kill_switch, get_fallback_response
from lib.bigquery_client import BigQueryClient
from copilots.code_copilot.copilot import CodeCopilot

app = FastAPI(title="Code Copilot", version="1.0.0")

_copilot = None
_bq_client = None

def get_copilot() -> CodeCopilot:
    """Initialize the AI client only when an enabled request needs it."""
    global _copilot
    if _copilot is None:
        _copilot = CodeCopilot()
    return _copilot

def get_bq_client() -> BigQueryClient:
    """Initialize telemetry lazily so module import stays testable."""
    global _bq_client
    if _bq_client is None:
        _bq_client = BigQueryClient()
    return _bq_client

class CodeRequest(BaseModel):
    """Request model for code analysis."""
    code: str
    workflow: Optional[str] = None
    corrections_needed: Optional[int] = None

class SuggestResponse(BaseModel):
    """Response model for suggestions."""
    result: dict
    latency_ms: Optional[int]
    tokens_in: Optional[int]
    tokens_out: Optional[int]
    ai_enabled: bool
    outcome: str
    event_id: Optional[str] = None

@app.get("/health")
async def health():
    """Health check endpoint."""
    return {"status": "healthy", "ai_enabled": is_ai_enabled()}

@app.post("/suggest", response_model=SuggestResponse)
async def suggest(
    code: Optional[str] = Form(None),
    file: Optional[UploadFile] = File(None),
    workflow: Optional[str] = Form(None),
    corrections_needed: Optional[int] = Form(None),
):
    """
    Get code analysis suggestions.
    
    Accepts either:
    - code: Direct code text
    - file: Uploaded code file
    """
    # Get code content
    code_content = code
    if file:
        code_content = await file.read()
        code_content = code_content.decode('utf-8')
    
    if not code_content:
        raise HTTPException(status_code=400, detail="Either 'code' or 'file' must be provided")
    
    input_bytes = len(code_content.encode('utf-8'))
    
    def analyze():
        return get_copilot().analyze_code(code_content)
    
    response = with_kill_switch(analyze, "code")
    
    telemetry = response["result"].get("telemetry", {})
    latency_ms = telemetry.get("latency_ms")
    tokens_in = telemetry.get("tokens_in", 0)
    tokens_out = telemetry.get("tokens_out", 0)
    
    cost_usd = None
    if tokens_in and tokens_out:
        # Gemini 2.5 Flash standard text rates used for this 2025 PoC.
        # Check current pricing before reusing this calculation.
        cost_usd = (
            tokens_in / 1_000_000 * 0.30
            + tokens_out / 1_000_000 * 2.50
        )
    
    event_id = None
    try:
        event_id = get_bq_client().log_event(
            copilot="code",
            workflow=workflow,
            freshness_tier="seconds",
            input_bytes=input_bytes,
            output_bytes=len(str(response["result"]).encode('utf-8')),
            tokens_in=tokens_in,
            tokens_out=tokens_out,
            latency_ms=latency_ms,
            ai_enabled=response["ai_enabled"],
            corrections_needed=corrections_needed,
            outcome=response["outcome"],
            cost_usd=cost_usd,
        )
    except Exception as e:
        print(f"Failed to log to BigQuery: {e}")
    
    return SuggestResponse(
        result=response["result"],
        latency_ms=latency_ms,
        tokens_in=tokens_in,
        tokens_out=tokens_out,
        ai_enabled=response["ai_enabled"],
        outcome=response["outcome"],
        event_id=event_id,
    )

if __name__ == "__main__":
    import uvicorn
    port = int(os.getenv("PORT", 8080))
    uvicorn.run(app, host="0.0.0.0", port=port)

File: copilots/code_copilot/copilot.py

"""
Code Copilot - AI-powered code analysis and suggestions.
"""
import sys
import os
from typing import Dict, List, Any

parent_dir = os.path.join(os.path.dirname(__file__), '../../')
sys.path.insert(0, parent_dir)

from lib.ai_client import get_ai_client
from lib.telemetry import Telemetry

class CodeCopilot:
    """Code Copilot for test suggestions and refactor hints."""
    
    def __init__(self):
        self.ai_client = get_ai_client()
        self.system_prompt = """You are a code analysis assistant. Analyze code and provide:
1. Test case suggestions (unit tests, edge cases)
2. Refactoring hints (code quality, performance, maintainability)
Be concise and actionable. Format output as JSON with keys: test_suggestions (list) and refactor_hints (list)."""
    
    def analyze_code(self, code: str) -> Dict[str, Any]:
        """
        Analyze code and provide suggestions.
        
        Args:
            code: Source code to analyze
        
        Returns:
            Dictionary with test_suggestions and refactor_hints
        """
        code_block = f"```python\n{code}\n```"
        prompt = f"""Analyze the following code and provide test suggestions and refactoring hints:

{code_block}

Provide your analysis in JSON format with:
- test_suggestions: array of test case descriptions
- refactor_hints: array of refactoring suggestions"""
        
        telemetry = Telemetry()
        with telemetry.track():
            response, token_usage = self.ai_client.generate(
                prompt=prompt,
                system_prompt=self.system_prompt,
                max_tokens=2000,
                temperature=0.7,
            )
            telemetry.set_tokens(token_usage["tokens_in"], token_usage["tokens_out"])
        
        try:
            import json
            if "```json" in response:
                json_start = response.find("```json") + 7
                json_end = response.find("```", json_start)
                response = response[json_start:json_end].strip()
            elif "```" in response:
                json_start = response.find("```") + 3
                json_end = response.find("```", json_start)
                response = response[json_start:json_end].strip()
            
            result = json.loads(response)
        except (json.JSONDecodeError, ValueError):
            result = {
                "test_suggestions": [response],
                "refactor_hints": [],
            }
        
        return {
            "test_suggestions": result.get("test_suggestions", []),
            "refactor_hints": result.get("refactor_hints", []),
            "telemetry": telemetry.to_dict(),
        }

BigQuery Client

File: lib/bigquery_client.py

"""
BigQuery client for logging events and metrics.
"""
import os
import uuid
from datetime import datetime
from typing import Optional

try:
    from google.cloud import bigquery
    from google.cloud.exceptions import NotFound
except ImportError:
    bigquery = None
    NotFound = None

class BigQueryClient:
    """Client for logging events to BigQuery."""
    
    def __init__(self, project_id: Optional[str] = None, dataset_id: str = "copilot_poc"):
        if bigquery is None:
            raise ImportError("google-cloud-bigquery not installed")
        
        self.project_id = project_id or os.getenv("GCP_PROJECT_ID")
        if not self.project_id:
            raise ValueError("GCP_PROJECT_ID not provided")
        
        self.dataset_id = dataset_id
        self.client = bigquery.Client(project=self.project_id)
        self._ensure_dataset_and_table()
    
    def _ensure_dataset_and_table(self):
        """Ensure dataset and table exist."""
        dataset_ref = self.client.dataset(self.dataset_id)
        
        try:
            self.client.get_dataset(dataset_ref)
        except NotFound:
            dataset = bigquery.Dataset(dataset_ref)
            dataset.location = "EU"
            dataset.description = "Dataset for storing copilot events and metrics"
            self.client.create_dataset(dataset)
        
        table_ref = dataset_ref.table("events")
        try:
            table = self.client.get_table(table_ref)
            if "cost_usd" not in {field.name for field in table.schema}:
                table.schema = [
                    *table.schema,
                    bigquery.SchemaField("cost_usd", "NUMERIC"),
                ]
                self.client.update_table(table, ["schema"])
        except NotFound:
            schema = [
                bigquery.SchemaField("event_id", "STRING", mode="REQUIRED"),
                bigquery.SchemaField("ts", "TIMESTAMP", mode="REQUIRED"),
                bigquery.SchemaField("copilot", "STRING", mode="REQUIRED"),
                bigquery.SchemaField("workflow", "STRING"),
                bigquery.SchemaField("freshness_tier", "STRING"),
                bigquery.SchemaField("input_bytes", "INT64"),
                bigquery.SchemaField("output_bytes", "INT64"),
                bigquery.SchemaField("tokens_in", "INT64"),
                bigquery.SchemaField("tokens_out", "INT64"),
                bigquery.SchemaField("latency_ms", "INT64"),
                bigquery.SchemaField("ai_enabled", "BOOL"),
                bigquery.SchemaField("corrections_needed", "INT64"),
                bigquery.SchemaField("outcome", "STRING"),
                bigquery.SchemaField("cost_usd", "NUMERIC"),
                bigquery.SchemaField("notes", "STRING"),
            ]
            table = bigquery.Table(table_ref, schema=schema)
            table.time_partitioning = bigquery.TimePartitioning(field="ts")
            table.clustering_fields = ["copilot", "outcome"]
            self.client.create_table(table)
    
    def log_event(
        self,
        copilot: str,
        workflow: Optional[str] = None,
        freshness_tier: str = "seconds",
        input_bytes: Optional[int] = None,
        output_bytes: Optional[int] = None,
        tokens_in: Optional[int] = None,
        tokens_out: Optional[int] = None,
        latency_ms: Optional[int] = None,
        ai_enabled: bool = True,
        corrections_needed: Optional[int] = None,
        outcome: str = "success",
        cost_usd: Optional[float] = None,
        notes: Optional[str] = None,
    ) -> str:
        """
        Log an event to BigQuery.
        
        Returns:
            event_id: Unique identifier for the logged event
        """
        event_id = str(uuid.uuid4())
        
        row = {
            "event_id": event_id,
            "ts": datetime.utcnow().isoformat(),
            "copilot": copilot,
            "workflow": workflow,
            "freshness_tier": freshness_tier,
            "input_bytes": input_bytes,
            "output_bytes": output_bytes,
            "tokens_in": tokens_in,
            "tokens_out": tokens_out,
            "latency_ms": latency_ms,
            "ai_enabled": ai_enabled,
            "corrections_needed": corrections_needed,
            "outcome": outcome,
            "cost_usd": cost_usd,
            "notes": notes,
        }
        
        table_ref = self.client.dataset(self.dataset_id).table("events")
        errors = self.client.insert_rows_json(table_ref, [row])
        
        if errors:
            raise Exception(f"Failed to insert row: {errors}")
        
        return event_id

Telemetry Library

File: lib/telemetry.py

"""
Telemetry tracking for latency and token usage.
"""
import time
from typing import Dict, Optional

class Telemetry:
    """Track latency and token usage for AI operations."""
    
    def __init__(self):
        self.start_time: Optional[float] = None
        self.end_time: Optional[float] = None
        self.tokens_in: int = 0
        self.tokens_out: int = 0
    
    def __enter__(self):
        self.start_time = time.time()
        return self
    
    def __exit__(self, exc_type, exc_val, exc_tb):
        self.end_time = time.time()
    
    def track(self):
        """Context manager for tracking latency."""
        return self
    
    def set_tokens(self, tokens_in: int, tokens_out: int):
        """Set token usage."""
        self.tokens_in = tokens_in
        self.tokens_out = tokens_out
    
    @property
    def latency_ms(self) -> Optional[int]:
        """Get latency in milliseconds."""
        if self.start_time and self.end_time:
            return int((self.end_time - self.start_time) * 1000)
        return None
    
    def to_dict(self) -> Dict[str, any]:
        """Convert to dictionary."""
        return {
            "latency_ms": self.latency_ms,
            "tokens_in": self.tokens_in,
            "tokens_out": self.tokens_out,
        }

Terraform Infrastructure

File: infrastructure/modules/cloud_run/main.tf

variable "project_id" {
  description = "GCP Project ID"
  type        = string
}

variable "region" {
  description = "GCP Region"
  type        = string
  default     = "europe-west4"
}

variable "service_name" {
  description = "Cloud Run service name"
  type        = string
}

variable "image" {
  description = "Container image URL"
  type        = string
}

variable "service_account_email" {
  description = "Service account email"
  type        = string
}

variable "environment_variables" {
  description = "Environment variables"
  type        = map(string)
  default     = {}
}

variable "min_instances" {
  description = "Minimum number of instances"
  type        = number
  default     = 0  # Scale to zero for cost optimization
}

variable "max_instances" {
  description = "Maximum number of instances"
  type        = number
  default     = 3  # Limit max instances for cost control
}

resource "google_cloud_run_service" "service" {
  name     = var.service_name
  location = var.region
  project  = var.project_id
  template {
    spec {
      service_account_name = var.service_account_email
      
      containers {
        image = var.image
        
        resources {
          limits = {
            cpu    = "1"
            memory = "512Mi"
          }
        }
        
        dynamic "env" {
          for_each = var.environment_variables
          content {
            name  = env.key
            value = env.value
          }
        }
      }
      
      container_concurrency = 80
      timeout_seconds       = 300
    }
    
    metadata {
      annotations = {
        "autoscaling.knative.dev/minScale" = tostring(var.min_instances)
        "autoscaling.knative.dev/maxScale" = tostring(var.max_instances)
      }
      
      labels = {
        environment = "dev"
        service     = var.service_name
      }
    }
  }
  traffic {
    percent         = 100
    latest_revision = true
  }
}

# Private access only (bearer token authentication)
resource "google_cloud_run_service_iam_member" "authenticated_access" {
  service  = google_cloud_run_service.service.name
  location = google_cloud_run_service.service.location
  role     = "roles/run.invoker"
  member   = "serviceAccount:${var.service_account_email}"
}

output "service_url" {
  description = "Cloud Run service URL"
  value       = google_cloud_run_service.service.status[0].url
}

BigQuery Schema

File: scripts/setup_bigquery.sql

-- BigQuery Setup Script for Copilot PoC
-- Dataset: copilot_poc

-- Create dataset
-- If copilot_poc already exists outside the EU, copy it to a new EU dataset;
-- CREATE SCHEMA IF NOT EXISTS does not change an existing dataset's location.
CREATE SCHEMA IF NOT EXISTS `copilot_poc`
OPTIONS(
  description="Dataset for storing copilot events and metrics",
  location="EU"
);

-- Events table
CREATE TABLE IF NOT EXISTS `copilot_poc.events` (
  event_id STRING NOT NULL,
  ts TIMESTAMP NOT NULL,
  copilot STRING NOT NULL,
  workflow STRING,
  freshness_tier STRING,
  input_bytes INT64,
  output_bytes INT64,
  tokens_in INT64,
  tokens_out INT64,
  latency_ms INT64,
  ai_enabled BOOL,
  corrections_needed INT64,
  outcome STRING,
  cost_usd NUMERIC,
  notes STRING
)
PARTITION BY DATE(ts)
CLUSTER BY copilot, outcome
OPTIONS(
  description="One row per copilot invocation with full telemetry"
);

-- Migration for datasets created by an earlier version of this PoC
ALTER TABLE `copilot_poc.events`
ADD COLUMN IF NOT EXISTS cost_usd NUMERIC;

-- Paired human task measurements used for time-saved and edit-reduction gates
CREATE TABLE IF NOT EXISTS `copilot_poc.experiment_trials` (
  trial_id STRING NOT NULL,
  measured_at TIMESTAMP NOT NULL,
  copilot STRING NOT NULL,
  baseline_task_ms INT64 NOT NULL,
  assisted_task_ms INT64 NOT NULL,
  baseline_corrections INT64,
  assisted_corrections INT64
)
PARTITION BY DATE(measured_at)
CLUSTER BY copilot;

-- Catalog table
CREATE TABLE IF NOT EXISTS `copilot_poc.catalog` (
  workflow STRING NOT NULL,
  copilot STRING NOT NULL,
  owner STRING,
  freshness_tier STRING,
  sla_ms INT64
)
OPTIONS(
  description="Workflow registry with freshness tiers and SLAs"
);

-- Upsert catalog entries
MERGE `copilot_poc.catalog` AS target
USING UNNEST([
  STRUCT('code-copilot-workflow' AS workflow, 'code' AS copilot, 'dev-team' AS owner, 'seconds' AS freshness_tier, 2000 AS sla_ms),
  STRUCT('reading-copilot-nightly' AS workflow, 'reading' AS copilot, 'dev-team' AS owner, 'daily' AS freshness_tier, 300000 AS sla_ms),
  STRUCT('meeting-copilot-workflow' AS workflow, 'meeting' AS copilot, 'dev-team' AS owner, 'minutes' AS freshness_tier, 180000 AS sla_ms)
]) AS source
ON target.workflow = source.workflow
WHEN MATCHED THEN
  UPDATE SET
    copilot = source.copilot,
    owner = source.owner,
    freshness_tier = source.freshness_tier,
    sla_ms = source.sla_ms
WHEN NOT MATCHED THEN
  INSERT (workflow, copilot, owner, freshness_tier, sla_ms)
  VALUES (source.workflow, source.copilot, source.owner, source.freshness_tier, source.sla_ms);

-- Metrics view (daily aggregates)
CREATE OR REPLACE VIEW `copilot_poc.metrics` AS
SELECT
  DATE(ts) as date,
  copilot,
  COUNT(*) as total_events,
  AVG(latency_ms) as avg_latency_ms,
  APPROX_QUANTILES(latency_ms, 100)[OFFSET(95)] as p95_latency_ms,
  SUM(tokens_in) as total_tokens_in,
  SUM(tokens_out) as total_tokens_out,
  SUM(cost_usd) as total_cost_usd,
  AVG(corrections_needed) as avg_corrections,
  COUNTIF(outcome = 'success') as success_count,
  COUNTIF(outcome = 'fallback') as fallback_count,
  COUNTIF(outcome = 'error') as error_count,
  COUNTIF(outcome = 'success') / COUNT(*) as success_rate
FROM
  `copilot_poc.events`
GROUP BY
  date, copilot
ORDER BY
  date DESC, copilot;

Testing Strategy

Unit Tests

File: tests/test_kill_switch.py

"""
Unit tests for kill switch functionality.
"""
import os
import pytest
from unittest.mock import patch, MagicMock
from lib.kill_switch import is_ai_enabled, get_fallback_response, with_kill_switch

def test_is_ai_enabled_true():
    """Test AI enabled when environment variable is true."""
    with patch.dict(os.environ, {"AI_ENABLED": "true"}):
        assert is_ai_enabled() is True

def test_is_ai_enabled_false():
    """Test AI disabled when environment variable is false."""
    with patch.dict(os.environ, {"AI_ENABLED": "false"}):
        assert is_ai_enabled() is False

def test_is_ai_enabled_default():
    """Test AI enabled by default when environment variable is not set."""
    with patch.dict(os.environ, {}, clear=True):
        assert is_ai_enabled() is True

def test_get_fallback_response_code():
    """Test fallback response for code copilot."""
    response = get_fallback_response("code")
    assert "test_suggestions" in response
    assert "refactor_hints" in response
    assert "message" in response
    assert response["test_suggestions"] == []
    assert response["refactor_hints"] == []

def test_get_fallback_response_reading():
    """Test fallback response for reading copilot."""
    response = get_fallback_response("reading")
    assert "digest" in response
    assert "follow_up_prompts" in response
    assert "message" in response

def test_with_kill_switch_disabled():
    """Test kill switch when AI is disabled."""
    with patch.dict(os.environ, {"AI_ENABLED": "false"}):
        mock_function = MagicMock()
        result = with_kill_switch(mock_function, "code")
        
        assert result["ai_enabled"] is False
        assert result["outcome"] == "fallback"
        assert "result" in result
        mock_function.assert_not_called()

def test_with_kill_switch_enabled_success():
    """Test kill switch when AI is enabled and succeeds."""
    with patch.dict(os.environ, {"AI_ENABLED": "true"}):
        mock_function = MagicMock(return_value={"test": "result"})
        result = with_kill_switch(mock_function, "code")
        
        assert result["ai_enabled"] is True
        assert result["outcome"] == "success"
        assert result["result"] == {"test": "result"}
        mock_function.assert_called_once()

def test_with_kill_switch_enabled_error():
    """Test kill switch when AI is enabled but fails."""
    with patch.dict(os.environ, {"AI_ENABLED": "true"}):
        mock_function = MagicMock(side_effect=Exception("AI error"))
        result = with_kill_switch(mock_function, "code")
        
        assert result["ai_enabled"] is True
        assert result["outcome"] == "error"
        assert "error" in result
        assert result["error"] == "AI error"

File: tests/test_ai_client.py

"""
Unit tests for AI client abstraction.
"""
import os
import pytest
from unittest.mock import patch, MagicMock
from lib.ai_client import get_ai_client, VertexAIClient, AIProvider

@patch('lib.ai_client.aiplatform')
@patch('lib.ai_client.GenerativeModel')
def test_vertex_ai_client_init(mock_model_class, mock_aiplatform):
    """Test Vertex AI client initialization."""
    mock_model = MagicMock()
    mock_model_class.return_value = mock_model
    
    with patch.dict(os.environ, {"GCP_PROJECT_ID": "test-project"}):
        client = VertexAIClient(
            project_id="test-project",
            location="europe-west4",
            model="gemini-2.5-flash"
        )
        
        assert client.project_id == "test-project"
        assert client.location == "europe-west4"
        assert client.model_name == "gemini-2.5-flash"
        mock_aiplatform.init.assert_called_once()
        mock_model_class.assert_called_once_with("gemini-2.5-flash")

@patch('lib.ai_client.aiplatform')
@patch('lib.ai_client.GenerativeModel')
def test_vertex_ai_client_generate(mock_model_class, mock_aiplatform):
    """Test Vertex AI client generate method."""
    mock_model = MagicMock()
    mock_response = MagicMock()
    mock_response.text = "Test response"
    mock_response.usage_metadata.prompt_token_count = 20
    mock_response.usage_metadata.candidates_token_count = 10
    mock_response.usage_metadata.thoughts_token_count = 5
    mock_model.generate_content.return_value = mock_response
    mock_model_class.return_value = mock_model
    
    with patch.dict(os.environ, {"GCP_PROJECT_ID": "test-project"}):
        client = VertexAIClient()
        response, token_usage = client.generate(
            prompt="Test prompt",
            system_prompt="System prompt",
            max_tokens=1000,
            temperature=0.7
        )
        
        assert response == "Test response"
        assert token_usage == {"tokens_in": 20, "tokens_out": 15}
        mock_model.generate_content.assert_called_once()

File: tests/test_code_copilot.py

"""
Unit tests for code copilot.
"""
import pytest
from unittest.mock import patch, MagicMock
from copilots.code_copilot.copilot import CodeCopilot

@patch('copilots.code_copilot.copilot.get_ai_client')
def test_code_copilot_analyze_code(mock_get_client):
    """Test code copilot analyze_code method."""
    mock_client = MagicMock()
    mock_client.generate.return_value = (
        '{"test_suggestions": ["Test 1", "Test 2"], "refactor_hints": ["Hint 1"]}',
        {"tokens_in": 100, "tokens_out": 50}
    )
    mock_get_client.return_value = mock_client
    
    copilot = CodeCopilot()
    result = copilot.analyze_code("def test(): pass")
    
    assert "test_suggestions" in result
    assert "refactor_hints" in result
    assert "telemetry" in result
    assert len(result["test_suggestions"]) == 2
    assert len(result["refactor_hints"]) == 1
    assert result["telemetry"]["tokens_in"] == 100
    assert result["telemetry"]["tokens_out"] == 50

Integration Tests

File: tests/integration/test_copilot_integration.py

"""
Integration tests for copilot services.
"""
import pytest
import os
from unittest.mock import patch
from fastapi.testclient import TestClient
from copilots.code_copilot.main import app

@pytest.fixture
def client():
    """Create test client."""
    return TestClient(app)

def test_health_endpoint(client):
    """Test health endpoint."""
    response = client.get("/health")
    assert response.status_code == 200
    data = response.json()
    assert "status" in data
    assert "ai_enabled" in data

@patch('copilots.code_copilot.main.get_bq_client')
@patch.dict(os.environ, {"AI_ENABLED": "false"})
def test_suggest_endpoint_kill_switch(mock_get_bq_client, client):
    """Test suggest endpoint with kill switch disabled."""
    mock_get_bq_client.return_value.log_event.return_value = "test-event"
    response = client.post(
        "/suggest",
        data={"code": "def test(): pass"}
    )
    assert response.status_code == 200
    data = response.json()
    assert data["ai_enabled"] is False
    assert data["outcome"] == "fallback"

Performance Tests

File: tests/performance/test_latency.py

"""
Performance tests for latency requirements.
"""
import pytest
import time
from copilots.code_copilot.copilot import CodeCopilot

def test_code_copilot_latency_seconds_tier():
    """Test code copilot meets seconds-tier latency requirement."""
    copilot = CodeCopilot()
    start_time = time.time()
    result = copilot.analyze_code("def test(): pass")
    elapsed_ms = (time.time() - start_time) * 1000
    
    # Seconds tier: p95 < 2000ms
    assert elapsed_ms < 2000, f"Latency {elapsed_ms}ms exceeds 2000ms SLA"
    assert "telemetry" in result
    assert result["telemetry"]["latency_ms"] < 2000

End-to-End Tests

File: tests/e2e/test_full_workflow.py

"""
End-to-end tests for full copilot workflow.
"""
import pytest
import requests
import os

@pytest.mark.e2e
def test_code_copilot_e2e():
    """Test full code copilot workflow end-to-end."""
    service_url = os.getenv("CODE_COPILOT_URL", "http://localhost:8080")
    
    # Test health check
    health_response = requests.get(f"{service_url}/health")
    assert health_response.status_code == 200
    
    # Test code analysis
    code = """
def calculate_total(items):
    total = 0
    for item in items:
        total += item.price
    return total
"""
    suggest_response = requests.post(
        f"{service_url}/suggest",
        data={"code": code}
    )
    assert suggest_response.status_code == 200
    data = suggest_response.json()
    assert "result" in data
    assert "event_id" in data

Measurement & Metrics

The time-saving and edit-reduction gates use paired manual task runs stored in experiment_trials. Model inference latency in events is a service metric; it is not treated as human time saved.

BigQuery Queries

Daily Metrics:

SELECT * FROM `copilot_poc.metrics`
WHERE date >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)
ORDER BY date DESC, copilot;

Time Saved Calculation:

WITH paired_trials AS (
  SELECT
    copilot,
    100 * SAFE_DIVIDE(
      baseline_task_ms - assisted_task_ms,
      baseline_task_ms
    ) AS time_saved_percent
  FROM `copilot_poc.experiment_trials`
  WHERE baseline_task_ms > 0
    AND DATE(measured_at) >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)
)
SELECT
  copilot,
  COUNT(*) AS paired_runs,
  APPROX_QUANTILES(time_saved_percent, 100)[OFFSET(50)]
    AS median_time_saved_percent
FROM paired_trials
GROUP BY copilot;

Cost per 1K Predictions:

SELECT
  copilot,
  COUNT(*) as total_predictions,
  SUM(cost_usd) as total_cost_usd,
  SUM(cost_usd) / COUNT(*) * 1000 as cost_per_1k_predictions_usd
FROM
  `copilot_poc.events`
WHERE
  outcome = 'success'
  AND cost_usd IS NOT NULL
  AND DATE(ts) >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)
GROUP BY
  copilot;

Kill Switch Effectiveness:

SELECT
  DATE(ts) as date,
  copilot,
  COUNTIF(outcome = 'fallback') as fallback_count,
  COUNTIF(outcome = 'success') as success_count,
  COUNTIF(outcome = 'fallback') / COUNT(*) * 100 as fallback_rate
FROM
  `copilot_poc.events`
WHERE
  DATE(ts) >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)
GROUP BY
  date, copilot
ORDER BY
  date DESC, copilot;

Quality Metrics (Edits Reduced):

WITH paired_trials AS (
  SELECT
    copilot,
    100 * SAFE_DIVIDE(
      baseline_corrections - assisted_corrections,
      baseline_corrections
    ) AS edits_reduced_percent
  FROM `copilot_poc.experiment_trials`
  WHERE baseline_corrections > 0
    AND assisted_corrections IS NOT NULL
    AND DATE(measured_at) >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)
)
SELECT
  copilot,
  COUNT(*) AS paired_runs,
  APPROX_QUANTILES(edits_reduced_percent, 100)[OFFSET(50)]
    AS median_edits_reduced_percent
FROM paired_trials
GROUP BY copilot;

Results

Reading Copilot

Hypothesis: H1 (time saved), H2 (edits reduced), H3 (cost)

Results:

• Time saved: 17% (median over 5 runs)
• Edits reduced: 21% vs. baseline
• Cost: $0.01/task (daily batch)
• p95 latency: 45 seconds (within daily tier SLA)
• Status: PASSED - Kept

Analysis:

Daily batch processing eliminated real-time costs while meeting the “fresh enough” requirement. The 10-bullet digest format reduced cognitive load and editing time.

Meeting Copilot

Hypothesis: H1 (time saved), H2 (edits reduced), H4 (latency)

Results:

• Time saved: 20% (median over 5 runs)
• Edits reduced: 22% vs. baseline
• p95 latency: 2.5 minutes (within minutes tier SLA)
• Cost: $0.02/task
• Status: PASSED - Kept

Analysis:

Structured action items with owners and deadlines reduced follow-up time. The minutes-tier freshness met the use case without requiring real-time processing.

Code Copilot

Hypothesis: H1 (time saved), H2 (edits reduced), H4 (latency)

Results:

• Time saved: 8% (median over 5 runs) - Below 15% threshold
• Edits reduced: 12% vs. baseline - Below 20% threshold
• p95 latency: 1.8 seconds (within seconds tier SLA)
• Cost: $0.005/task
• Status: FAILED - Retired

Analysis:

While latency was acceptable, the time saved and edit reduction didn’t meet the success gates. The copilot was helpful but not transformative enough to justify keeping. Future experiments with on-device models or better prompts may improve results.

Kill Switch

Hypothesis: H5 (reversibility)

Results:

• Fallback success rate: 100% (10/10 tests)
• Response time: <50ms (deterministic fallback)
• Logging: All fallback events logged to BigQuery
• Status: PASSED

Analysis:

The kill switch provided clean reversibility with zero AI API calls when disabled. All fallback events were logged, enabling audit trails.

Governance & Trust

Model Card: AI Copilots PoC

Model Details

• Provider: Google Vertex AI (Gemini 2.5 Flash)
• Locations: europe-west4 for compute; EU multi-region for BigQuery
• Use Case: Personal workflow assistance (reading, meetings, code)

Intended Use

• Reading Copilot: Generate 10-bullet digests from documents
• Meeting Copilot: Extract action items with owners and deadlines
• Code Copilot: Suggest test cases and refactoring hints

Limitations

• No business data or PII processing
• Personal use only
• Outputs require human review
• No guarantees on accuracy or completeness

Failure Modes

• API outages -> Error fallback; operator can activate the kill switch
• Cost spikes -> Budget alert; operator can activate the kill switch
• Quality regression -> Manual review and kill switch if needed

Bias & Safety

• Prompts designed to avoid absolute claims
• Citations encouraged when possible
• Human-in-the-loop for all outputs

Bias Checklist

Pre-Deployment

• Prompts reviewed for bias
• Output formats structured to reduce interpretation errors
• Kill switch tested and documented
• Audit logs enabled

During Operation

• Monitor for quality regression
• Track corrections_needed metric
• Review fallback events
• Cost monitoring alerts

Post-Deployment

• Review outcomes weekly
• Compare against baseline metrics
• Document failures and learnings

What Failed & Why

Code Copilot Retirement

Why it failed:

• Time saved (8%) below 15% threshold
• Edits reduced (12%) below 20% threshold
• Helpful but not transformative

Lessons:

• Not all tasks need generative AI
• Right-time beats real-time (reading copilot succeeded with daily batch)
• Small scope wins (easier to measure and decide)

Next steps:

• Experiment with on-device models for lower latency/privacy
• Try different prompts or fine-tuning
• Consider rule-based alternatives for code analysis

Reusable Pattern

One-Page Playbook

1. Start Tiny

• One copilot, one endpoint, one metric table
• Measure baseline honestly (5+ runs)

2. Map Freshness to Decisions

• Seconds: User waiting (interactive)
• Minutes: Acceptable delay (post-processing)
• Daily: No user waiting (batch)

3. Add Kill Switch

• Environment variable: AI_ENABLED=false
• Deterministic fallback responses
• Log all fallback events

4. Measure Strictly

• Time saved (median over 5+ runs)
• Edits reduced vs. baseline
• Cost per task
• p95 latency within SLA

5. Keep Only What Beats Control

• ≥15% time saved
• ≥20% fewer edits
• Stable cost
• Stop the rest

Conclusion

This PoC proved that small, measurable AI helpers can deliver real value when:

• Freshness is mapped to decisions (right-time, not real-time)
• Success gates are strict (≥15% time saved, ≥20% fewer edits)
• A kill switch provides reversibility
• Cost is predictable and monitored

Two copilots passed, one failed and was retired. The pattern is portable, governed, and repeatable - ready to template for other workflows.

Author: Dr. Atabak Kheirkhah
Date: November 17, 2025
Contact: atabakkheirkhah@gmail.com

Most AI projects fail before the model is the problem. They fail because the platform underneath cannot answer freshness, ownership, lineage, or trust. This is the scorecard I keep coming back to when serious AI or agent work is on the table.

Context: By autumn 2025 enterprise AI had moved from “build a demo” toward “run in production”. Agents and RAG stacks were no longer exotic. In programs I touched, the bottleneck was rarely the model. It was whether the platform could feed, govern, and measure AI reliably.

When nobody could say when a table was last updated, who owned it, or whether yesterday’s numbers reconciled with finance, adding a vector database or an agent framework did not fix that. It amplified the mess.

Notes from migration work, batch scoring pipelines, and personal lab experiments on public cloud - not from any employer context.

The pattern I kept seeing

1. Leadership asks for a copilot, recommender, or agent that “uses our data”
2. Team spins up Vertex AI, OpenAI, or an internal gateway quickly
3. Demo works on a curated sample
4. Production breaks on freshness, permissions, definitions, lineage, cost, or silent pipeline failures from three weeks ago

The model was never the first failure. The platform was. AI exposes weak data platforms faster than BI ever did.

What “AI-ready” meant in practice

Not “we bought a lakehouse” or “we deployed an LLM”.

It meant the platform could consistently provide:

If you cannot do this for a batch report, you will not do it for production AI.

Five foundations that had to move first

1. Layered architecture. Landing -> curated -> serving -> feature/context. Red flag: every AI project rebuilds “customer 360” from scratch. Publish serving tables and contracts for entities AI will touch.

2. Quality with teeth. Freshness SLAs, reconciliation, schema drift detection, quarantine - not a dashboard that says 98% complete while the missing 2% is revenue or consent.

3. Orchestration you can explain. One source of truth per domain, explicit dependencies, idempotent partitions, run metadata. Honest freshness tiers: seconds / minutes / daily - not fake real-time everywhere.

4. Ownership, lineage, access. Business owner, technical owner, access class. Least privilege for humans, services, and agents. An agent with broad warehouse access and no approval flow is an exfiltration pattern waiting for a prompt.

5. Cost as design. Partition for real filters. Materialize heavy joins. Budgets and kill switches. AI makes expensive mistakes show up weekly, not quarterly.

Scorecard I still use

Architecture: layered paths · serving matches access patterns · versioned feature/context
Quality: freshness · reconciliation · drift/quarantine
Operations: real dependency graph · idempotent partitions · runbooks with owners
Governance: named owners · lineage to model input · least privilege
Economics: cost-reviewed patterns · freshness tied to schedules · budgets and kill switches

12-15 -> production-ready. 8-11 -> bounded pilots. Below 8 -> fix the platform first.

I kept a rough score sheet as YAML for intake reviews (personal notes / lab style):

use_case: support_case_summary
datasets:
  - name: curated.cases
    freshness_sla_min: 60
    owner_business: support-ops
    owner_technical: data-platform
    score_architecture: 2   # 0-3
    score_quality: 2
    score_ops: 1
    score_governance: 2
    score_economics: 2
total: 9   # bounded pilot only until ops improves

First 90 days when recovery was possible

Weeks 1-4: Inventory AI-critical datasets; freshness and ownership metadata; draw the real pipeline graph; reconcile top revenue/compliance tables.

Weeks 5-8: Kill duplicate pipelines; publish serving tables; drift checks; access classes.

Weeks 9-12: Version feature tables; audit automated retrieval; cost ceilings; contracts for agent entities.

From the leadership seat

When leadership asked to “do AI” this quarter:

I would not approve model spend before a platform scorecard on the datasets in scope. Below 8 -> bounded pilot only, with a named platform track in parallel.

I tried to split capacity: a real share of the initiative for ownership, contracts, freshness, access - packaged as part of the AI work so it did not get deprioritized as “platform”.

Kill criteria upfront - time saved, override rate, cost ceiling, reversibility.

One executive for definitions when marketing, finance, and product disagree on “customer” or “revenue”. That is not an LLM problem.

Protect the best engineer from hero mode. If one person is the only one who understands the graph, you are person-dependent, not AI-ready.

Closing

Production AI needs a platform that can answer: what data was used, how fresh, who owns it, what happens when it breaks, what it cost. Those answers had to come first.

Problem: Either you pay too much (min too high) or you get tail latency (min too low). Balance it.

Cloud Run’s serverless model is powerful, but misconfigured concurrency and instance settings can lead to either excessive costs or poor user experience. This guide walks through practical tuning strategies based on real production workloads.

Understanding Cloud Run’s Scaling Model

Cloud Run scales instances based on request concurrency-the number of simultaneous requests each container instance handles. Unlike traditional autoscaling that uses CPU/memory metrics, Cloud Run scales when:

1. All instances are at capacity (concurrency limit reached)
2. New requests arrive (queue forms)
3. Traffic increases (more instances needed)

The key insight: concurrency is your primary cost lever. Higher concurrency = fewer instances = lower cost, but only if your application can handle it without degrading latency.

Settings that Matter

1. Concurrency: The Primary Cost Control

Concurrency determines how many requests each instance handles simultaneously. This is your biggest cost optimization opportunity.

Starting Points:

• I/O-bound services (APIs, web servers): Start with 40-80 concurrent requests
• CPU-bound services (ML inference, data processing): Start with 4-8 concurrent requests
• Mixed workloads: Start with 20-40 and tune based on metrics

Why it matters:

• Concurrency of 1 = every request gets its own instance (expensive, fast)
• Concurrency of 1000 = one instance handles everything (cheap, but may queue)

Real-world example:

# High-traffic API (I/O-bound, database queries)
gcloud run deploy api \
  --concurrency=80 \
  --cpu=2 \
  --memory=2Gi

# ML inference service (CPU-bound, model predictions)
gcloud run deploy ml-service \
  --concurrency=4 \
  --cpu=4 \
  --memory=8Gi

Tuning strategy:

1. Start conservative (lower concurrency)
2. Monitor p95/p99 latency during peak traffic
3. Gradually increase concurrency until latency degrades
4. Back off 20% from the breaking point

2. Min Instances: Eliminating Cold Starts

Min instances keep containers warm, eliminating cold start latency for critical paths.

When to use:

• User-facing APIs with strict latency SLAs (< 200ms p95)
• High-traffic endpoints that can’t tolerate cold starts
• Business-critical services during peak hours

Cost trade-off:

• Min instances = 0: Pay only for requests, but cold starts on first request
• Min instances = 5: Always paying for 5 instances, but zero cold starts

Practical guidance:

• Critical APIs: 1-3 min instances
• Background jobs: 0 min instances (cold starts acceptable)
• Peak hours only: Use Cloud Scheduler to adjust min instances by time

Example: Business hours only

# During business hours (9 AM - 6 PM)
gcloud run services update api \
  --min-instances=3 \
  --region=europe-west3

# After hours (via Cloud Scheduler)
gcloud run services update api \
  --min-instances=0 \
  --region=europe-west3