[{"data":1,"prerenderedAt":1197},["ShallowReactive",2],{"/blog/reducing-pager-fatigue-and-improving-on-call-life":3,"navigation-en-us":410,"banner-en-us":838,"footer-en-us":848,"blog-post-authors-en-us-Steve Azzopardi":1093,"blog-related-posts-en-us-reducing-pager-fatigue-and-improving-on-call-life":1107,"blog-promotions-en-us":1133,"next-steps-en-us":1187},{"id":4,"title":5,"authors":6,"body":8,"category":390,"date":391,"description":392,"extension":393,"externalUrl":394,"faq":394,"featured":395,"heroImage":396,"meta":397,"navigation":398,"path":399,"seo":400,"slug":404,"stem":405,"tags":406,"template":408,"updatedDate":394,"__hash__":409},"blogPosts/en-us/blog/reducing-pager-fatigue-and-improving-on-call-life.md","How we improved on-call life by reducing pager noise",[7],"Steve Azzopardi",{"type":9,"value":10,"toc":384},"minimark",[11,28,94,102,109,112,118,127,132,150,153,163,177,183,192,198,201,205,219,244,250,271,274,287,294,327,346,350,353,364,370],[12,13,14,15,21,22,27],"p",{},"To monitor the health of GitLab.com we use multiple\n",[16,17,20],"a",{"href":18,"rel":19},"https://sre.google/workbook/implementing-slos/#what-to-measure-using-slis",[],"SLIs","\nfor each service. We then page the on-call when one of these SLIs is not\nmeeting our internal ",[16,23,26],{"href":24,"rel":25},"https://sre.google/workbook/implementing-slos/#decision-making-using-slos-and-error-budgets",[],"SLOs and burning through the error\nbudget","\nwith the hopes of fixing the problem before too many of our users even notice.",[12,29,30,31,36,37,42,43,51,52,57,58,63,64,69,70,75,76,81,82,87,88,93],{},"All of our services SLIs and SLOs are defined using ",[16,32,35],{"href":33,"rel":34},"https://jsonnet.org/",[],"jsonnet"," in\nwhat we call the ",[16,38,41],{"href":39,"rel":40},"https://gitlab.com/gitlab-com/runbooks/-/tree/168397dee0ad955bfb473fd0ddb9146667eeaa13/metrics-catalog",[],"metrics-catalog","\nwhere we specify a service and its SLIs/SLOs. For example, the ",[16,44,47],{"href":45,"rel":46},"https://gitlab.com/gitlab-com/runbooks/-/blob/168397dee0ad955bfb473fd0ddb9146667eeaa13/metrics-catalog/services/web-pages.jsonnet",[],[48,49,50],"code",{},"web-pages","\nservice has ",[16,53,56],{"href":54,"rel":55},"https://gitlab.com/gitlab-com/runbooks/-/blob/168397dee0ad955bfb473fd0ddb9146667eeaa13/metrics-catalog/services/web-pages.jsonnet#L22",[],"an apdex SLO of 99.5%","\nand multiple SLIs such as ",[16,59,62],{"href":60,"rel":61},"https://gitlab.com/gitlab-com/runbooks/-/blob/168397dee0ad955bfb473fd0ddb9146667eeaa13/metrics-catalog/services/web-pages.jsonnet#L60",[],"loadbalancer",",\n",[16,65,68],{"href":66,"rel":67},"https://gitlab.com/gitlab-com/runbooks/-/blob/168397dee0ad955bfb473fd0ddb9146667eeaa13/metrics-catalog/services/web-pages.jsonnet#L80",[],"go server",",\nand ",[16,71,74],{"href":72,"rel":73},"https://gitlab.com/gitlab-com/runbooks/-/blob/168397dee0ad955bfb473fd0ddb9146667eeaa13/metrics-catalog/services/web-pages.jsonnet#L113",[],"time to write HTTP headers",".\nHaving these in code we can automatically generate Prometheus ",[16,77,80],{"href":78,"rel":79},"https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/",[],"recording rules","\nand ",[16,83,86],{"href":84,"rel":85},"https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/",[],"alerting rules","\nfollowing ",[16,89,92],{"href":90,"rel":91},"https://sre.google/workbook/alerting-on-slos/#5-multiple-burn-rate-alerts",[],"multiple burn rate alerts",".\nEvery time we start burning through our 30-day error budget for an SLI too fast\nwe page the SRE on-call to investigate and solve the problem.",[12,95,96,97,101],{},"This setup has been working well for us for over two years now, but one big\npain point remained when there was a service-wide degradation. The SRE on-call\nwas getting paged ",[98,99,100],"strong",{},"for every SLI"," associated with a service or its\ndownstream dependencies, meaning they can get up to 10 pages per service since\nthe service has 3-5 SLIs on average and we also have regional and canary SLIs.\nThis gets very distracting, it's stress-inducing, and it also doesn't let the\non-call focus on solving the problem but just acknowledges pages. For example\nbelow we can see the on-call getting paged 11 times in 5 minutes for the same\nservice.",[12,103,104],{},[105,106],"img",{"alt":107,"src":108},"web-pages alert storm","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782398914/blog/Content%20Images/reducing-pager-fatigue/web-pages-alert-storm.png",[12,110,111],{},"What is even worse is when we have a site-wide outage, where the on-call can\nend up getting 50+ pages because all services are in a degraded state.",[12,113,114],{},[105,115],{"alt":116,"src":117},"site wide outage alert storm","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782398914/blog/Content%20Images/reducing-pager-fatigue/site-wide-outage-alert-storm.png",[12,119,120,121,126],{},"It was a big problem for the quality of life for the on-call and we needed to\nfix this. We started doing some research on how to best solve this problem and\nopened an issue to ",[16,122,125],{"href":123,"rel":124},"https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15721",[],"document all possible\nsolutions",".\nAfter some time we decided to go with grouping alerts by service and\nintroducing service dependencies for alerting/paging.",[128,129,131],"h2",{"id":130},"group-alerts-by-service","Group alerts by service",[12,133,134,135,137,138,143,144,149],{},"The smallest and most effective iteration was to group the alerts by the\nservice. Taking the previous example where the ",[48,136,50],{}," service paged the\non-call 11 times, it should have only paged the on-call once, and shown\nwhich SLIs were affected. We use ",[16,139,142],{"href":140,"rel":141},"https://prometheus.io/docs/alerting/latest/alertmanager/",[],"Alertmanager"," for\nall our alerting logic, and this already had a feature called\n",[16,145,148],{"href":146,"rel":147},"https://prometheus.io/docs/alerting/latest/alertmanager/#grouping",[],"grouping","\nso we could group alerts by labels.",[12,151,152],{},"This is what an alert looks like in our Prometheus setup:",[154,155,161],"pre",{"className":156,"code":158,"language":159,"meta":160},[157],"language-text","ALERTS{aggregation=\"regional_component\", alert_class=\"slo_violation\", alert_type=\"symptom\", alertname=\"WebPagesServiceServerApdexSLOViolationRegional\", alertstate=\"firing\", component=\"server\", env=\"gprd\", environment=\"gprd\", feature_category=\"pages\", monitor=\"global\", pager=\"pagerduty\", region=\"us-east1-d\", rules_domain=\"general\", severity=\"s2\", sli_type=\"apdex\", slo_alert=\"yes\", stage=\"main\", tier=\"sv\", type=\"web-pages\", user_impacting=\"yes\", window=\"1h\"}\n","text","",[48,162,158],{"__ignoreMap":160},[12,164,165,166,169,170,173,174,176],{},"All alerts have the ",[48,167,168],{},"type"," label attached to them to specify which service they\nbelong to. We can use this label and the ",[48,171,172],{},"env"," label to group all the\nproduction alerts that are firing for the ",[48,175,50],{}," service.",[12,178,179],{},[105,180],{"alt":181,"src":182},"grouping alerts by the type and env labels","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782398910/blog/Content%20Images/reducing-pager-fatigue/group-alerts-by-service.png",[12,184,185,186,191],{},"We also had to update our Pagerduty and Slack templates to show the right\ninformation. Before we only showed the alert title and description but this had\nto change since we are now alerting by service rather than by 1 specific SLO.\nYou can see the changes at ",[16,187,190],{"href":188,"rel":189},"https://gitlab.com/gitlab-com/runbooks/-/merge_requests/4684",[],"runbooks!4684",".",[12,193,194],{},[105,195],{"alt":196,"src":197},"Before and after on pages","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782398909/blog/Content%20Images/reducing-pager-fatigue/before-after-slack-alerts.png",[12,199,200],{},"This was already a big win! The on-call now gets a page saying \"service\nweb-pages\" and then the list of SLIs that are burning through the error budget - we went from 11 pages to 1 page!",[128,202,204],{"id":203},"service-dependencies","Service Dependencies",[12,206,207,208,63,211,214,215,218],{},"However we still had the problem that when a downstream service (such as the database)\nstarts burning through the error budget, it has a cascading effect where ",[48,209,210],{},"web",[48,212,213],{},"git",", and ",[48,216,217],{},"api"," will also start burning through the error budget and page the\non-call for each service. That was the next thing that we had to solve.",[12,220,221,222,224,225,228,229,231,232,237,238,240,241,243],{},"We needed some way to not alert on the ",[48,223,217],{}," service if the ",[48,226,227],{},"patroni","\n(database) service was burning through the error budget because it's clear if the\ndatabase is degraded the ",[48,230,217],{}," service will end up degraded as well. We used\nanother feature of Alertmanager called\n",[16,233,236],{"href":234,"rel":235},"https://prometheus.io/docs/alerting/latest/alertmanager/#inhibition",[],"inhibition","\nwhere we can tell Alertmanager to not alert on ",[48,239,217],{}," if some alerts on ",[48,242,227],{},"\nare already firing.",[12,245,246],{},[105,247],{"alt":248,"src":249},"visualization of how inhibit rules work","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782398910/blog/Content%20Images/reducing-pager-fatigue/inhibit-alerts.png",[12,251,252,253,256,257,262,263,270],{},"I've mentioned that all of our SLIs/SLOs are inside of the\n",[16,254,41],{"href":39,"rel":255},[],"\nso it was a natural fit to add dependencies there, and this is exactly what\nwe did in ",[16,258,261],{"href":259,"rel":260},"https://gitlab.com/gitlab-com/runbooks/-/merge_requests/4710",[],"runbooks!4710",". With this\nwe can specify that an SLI depends on another SLI of a different service which\nwill automatically create\n",[16,264,267],{"href":265,"rel":266},"https://prometheus.io/docs/alerting/latest/configuration/#inhibit_rule",[],[48,268,269],{},"inhibit_rules","\nfor Alertmanager.",[12,272,273],{},"Since inhibit rules could potentially prevent alerting someone, we've used\nthese sparingly. To avoid creating inhibit rules too broadly, we've implemented\nthe following restrictions:",[275,276,277,281,284],"ol",{},[278,279,280],"li",{},"An SLI can't depend on an SLI of the same service.",[278,282,283],{},"The SLI has to exist for that service.",[278,285,286],{},"We only allow equal operations, no regex on SLIs.",[12,288,289,290,293],{},"After that it was only a matter of adding the ",[48,291,292],{},"dependsOn"," on each service for example:",[275,295,296,307,317],{},[278,297,298],{},[16,299,302,304,305],{"href":300,"rel":301},"https://gitlab.com/gitlab-com/runbooks/-/merge_requests/4735",[],[48,303,210],{}," depends on ",[48,306,227],{},[278,308,309],{},[16,310,313,304,315],{"href":311,"rel":312},"https://gitlab.com/gitlab-com/runbooks/-/merge_requests/4750",[],[48,314,217],{},[48,316,227],{},[278,318,319],{},[16,320,323,304,325],{"href":321,"rel":322},"https://gitlab.com/gitlab-com/runbooks/-/merge_requests/4813",[],[48,324,50],{},[48,326,217],{},[12,328,329,330,332,333,336,337,339,340,342,343,345],{},"The ",[48,331,50],{}," inhibit rule shows a chain of dependencies from ",[48,334,335],{},"web-pages -> api -> patroni",", so if ",[48,338,227],{}," is burning through the error budget it will\nnot page for ",[48,341,217],{}," and ",[48,344,50],{}," services anymore!",[128,347,349],{"id":348},"how-its-working","How it's working",[12,351,352],{},"We have been using alert grouping and service dependencies for over a month now, and we have already seen some improvements:",[275,354,355,358,361],{},[278,356,357],{},"The on-call only gets paged once per service.",[278,359,360],{},"When there is a large site-wide outage they only get paged 5-10 times since we have external probes that also alert us.",[278,362,363],{},"There is an overall downward trend on pages for the on-call as seen below.",[12,365,366],{},[105,367],{"alt":368,"src":369},"pages trend","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782398913/blog/Content%20Images/reducing-pager-fatigue/pages-trend.png",[12,371,372,373,378,379],{},"Cover image by ",[16,374,377],{"href":375,"rel":376},"https://unsplash.com/@yaoqiqiqilai",[],"Yaoqi"," on ",[16,380,383],{"href":381,"rel":382},"https://unsplash.com/photos/7iatBuqFvY0",[],"Unsplash",{"title":160,"searchDepth":385,"depth":385,"links":386},2,[387,388,389],{"id":130,"depth":385,"text":131},{"id":203,"depth":385,"text":204},{"id":348,"depth":385,"text":349},"engineering","2022-07-19","Too many pages? Here's how we tackled on-call SRE quality of life by grouping alerts by service and only paging on downstream services.","md",null,false,"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749682368/Blog/Hero%20Images/cover.png",{},true,"/en-us/blog/reducing-pager-fatigue-and-improving-on-call-life",{"title":5,"description":392,"ogTitle":5,"ogDescription":392,"noIndex":395,"ogImage":396,"ogUrl":401,"ogSiteName":402,"ogType":403,"canonicalUrls":401},"https://about.gitlab.com/blog/reducing-pager-fatigue-and-improving-on-call-life","https://about.gitlab.com","article","reducing-pager-fatigue-and-improving-on-call-life","en-us/blog/reducing-pager-fatigue-and-improving-on-call-life",[407],"production","BlogPost","a9-rchgqBx7K-uc3U3SdbooSeQLT7l7yINxfv8RFkP8",{"logo":411,"freeTrial":416,"sales":421,"login":426,"items":431,"search":758,"minimal":789,"duo":808,"switchNav":817,"pricingDeployment":828},{"config":412},{"href":413,"dataGaName":414,"dataGaLocation":415},"/","gitlab logo","header",{"text":417,"config":418},"Get free trial",{"href":419,"dataGaName":420,"dataGaLocation":415},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":422,"config":423},"Request a demo",{"href":424,"dataGaName":425,"dataGaLocation":415},"/sales/?contact-topic=request-demo","sales",{"text":427,"config":428},"Sign in",{"href":429,"dataGaName":430,"dataGaLocation":415},"https://gitlab.com/users/sign_in/","sign in",[432,461,561,566,680,736],{"text":433,"config":434,"menu":436},"Platform",{"dataNavLevelOne":435},"platform",{"type":437,"columns":438},"cards",[439,445,453],{"title":433,"description":440,"link":441},"The intelligent orchestration platform for DevSecOps",{"text":442,"config":443},"Explore our Platform",{"href":444,"dataGaName":435,"dataGaLocation":415},"/platform/",{"title":446,"description":447,"link":448},"GitLab Duo Agent Platform","Agentic AI for the entire software lifecycle",{"text":449,"config":450},"Meet GitLab Duo",{"href":451,"dataGaName":452,"dataGaLocation":415},"/gitlab-duo-agent-platform/","gitlab duo agent platform",{"title":454,"description":455,"link":456},"Why GitLab","See the top reasons enterprises choose GitLab",{"text":457,"config":458},"Learn more",{"href":459,"dataGaName":460,"dataGaLocation":415},"/why-gitlab/","why gitlab",{"text":462,"left":398,"config":463,"menu":465},"Product",{"dataNavLevelOne":464},"solutions",{"type":466,"link":467,"columns":471,"feature":540},"lists",{"text":468,"config":469},"View all Solutions",{"href":470,"dataGaName":464,"dataGaLocation":415},"/solutions/",[472,496,519],{"title":473,"description":474,"link":475,"items":480},"Automation","CI/CD and automation to accelerate deployment",{"config":476},{"icon":477,"href":478,"dataGaName":479,"dataGaLocation":415},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[481,485,488,492],{"text":482,"config":483},"CI/CD",{"href":484,"dataGaLocation":415,"dataGaName":482},"/solutions/continuous-integration/",{"text":446,"config":486},{"href":451,"dataGaLocation":415,"dataGaName":487},"gitlab duo agent platform - product menu",{"text":489,"config":490},"Source Code Management",{"href":491,"dataGaLocation":415,"dataGaName":489},"/solutions/source-code-management/",{"text":493,"config":494},"Automated Software Delivery",{"href":478,"dataGaLocation":415,"dataGaName":495},"Automated software delivery",{"title":497,"description":498,"link":499,"items":504},"Security","Deliver code faster without compromising security",{"config":500},{"href":501,"dataGaName":502,"dataGaLocation":415,"icon":503},"/solutions/application-security-testing/","security and compliance","ShieldCheckLight",[505,509,514],{"text":506,"config":507},"Application Security Testing",{"href":501,"dataGaName":508,"dataGaLocation":415},"Application security testing",{"text":510,"config":511},"Software Supply Chain Security",{"href":512,"dataGaLocation":415,"dataGaName":513},"/solutions/supply-chain/","Software supply chain security",{"text":515,"config":516},"Software Compliance",{"href":517,"dataGaName":518,"dataGaLocation":415},"/solutions/software-compliance/","software compliance",{"title":520,"link":521,"items":526},"Measurement",{"config":522},{"icon":523,"href":524,"dataGaName":525,"dataGaLocation":415},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[527,531,535],{"text":528,"config":529},"Visibility & Measurement",{"href":524,"dataGaLocation":415,"dataGaName":530},"Visibility and Measurement",{"text":532,"config":533},"Value Stream Management",{"href":534,"dataGaLocation":415,"dataGaName":532},"/solutions/value-stream-management/",{"text":536,"config":537},"Analytics & Insights",{"href":538,"dataGaLocation":415,"dataGaName":539},"/solutions/analytics-and-insights/","Analytics and insights",{"title":541,"type":466,"items":542},"GitLab for",[543,549,555],{"text":544,"config":545},"Enterprise",{"icon":546,"href":547,"dataGaLocation":415,"dataGaName":548},"Building","/enterprise/","enterprise",{"text":550,"config":551},"Small Business",{"icon":552,"href":553,"dataGaLocation":415,"dataGaName":554},"Work","/small-business/","small business",{"text":556,"config":557},"Public Sector",{"icon":558,"href":559,"dataGaLocation":415,"dataGaName":560},"Organization","/solutions/public-sector/","public sector",{"text":562,"config":563},"Pricing",{"href":564,"dataGaName":565,"dataGaLocation":415,"dataNavLevelOne":565},"/pricing/","pricing",{"text":567,"config":568,"menu":570},"Resources",{"dataNavLevelOne":569},"resources",{"type":466,"link":571,"columns":575,"feature":669},{"text":572,"config":573},"View all resources",{"href":574,"dataGaName":569,"dataGaLocation":415},"/resources/",[576,609,636],{"title":577,"items":578},"Getting started",[579,584,589,594,599,604],{"text":580,"config":581},"Install",{"href":582,"dataGaName":583,"dataGaLocation":415},"/install/","install",{"text":585,"config":586},"Quick start guides",{"href":587,"dataGaName":588,"dataGaLocation":415},"/get-started/","quick setup checklists",{"text":590,"config":591},"Learn",{"href":592,"dataGaLocation":415,"dataGaName":593},"https://university.gitlab.com/","learn",{"text":595,"config":596},"Product documentation",{"href":597,"dataGaName":598,"dataGaLocation":415},"https://docs.gitlab.com/","product documentation",{"text":600,"config":601},"Best practice videos",{"href":602,"dataGaName":603,"dataGaLocation":415},"/getting-started-videos/","best practice videos",{"text":605,"config":606},"Integrations",{"href":607,"dataGaName":608,"dataGaLocation":415},"/integrations/","integrations",{"title":610,"items":611},"Discover",[612,617,622,627,631],{"text":613,"config":614},"Customer success stories",{"href":615,"dataGaName":616,"dataGaLocation":415},"/customers/","customer success stories",{"text":618,"config":619},"Blog",{"href":620,"dataGaName":621,"dataGaLocation":415},"/blog/","blog",{"text":623,"config":624},"Demo Hub",{"href":625,"dataGaName":626,"dataGaLocation":415},"/demo-hub/","demo hub",{"text":628,"config":629},"The Source",{"href":630,"dataGaName":621,"dataGaLocation":415},"/the-source/",{"text":632,"config":633},"Remote",{"href":634,"dataGaName":635,"dataGaLocation":415},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"title":637,"items":638},"Connect",[639,644,649,654,659,664],{"text":640,"config":641},"GitLab Services",{"href":642,"dataGaName":643,"dataGaLocation":415},"/services/","services",{"text":645,"config":646},"Contribute",{"href":647,"dataGaName":648,"dataGaLocation":415},"https://contributors.gitlab.com","contribute",{"text":650,"config":651},"Community",{"href":652,"dataGaName":653,"dataGaLocation":415},"/community/","community",{"text":655,"config":656},"Forum",{"href":657,"dataGaName":658,"dataGaLocation":415},"https://forum.gitlab.com/","forum",{"text":660,"config":661},"Events",{"href":662,"dataGaName":663,"dataGaLocation":415},"/events/","events",{"text":665,"config":666},"Partners",{"href":667,"dataGaName":668,"dataGaLocation":415},"/partners/","partners",{"config":670,"title":673,"text":674,"link":675},{"background":671,"textColor":672},"url('https://res.cloudinary.com/about-gitlab-com/image/upload/v1777322348/qpq8yrgn8knii57omj0c.png')","#000","What’s new in GitLab","Stay updated with our latest features and improvements.",{"text":676,"config":677},"Read the latest",{"href":678,"dataGaName":679,"dataGaLocation":415},"/whats-new/","whats new",{"text":681,"config":682,"menu":684},"Company",{"dataNavLevelOne":683},"company",{"type":466,"columns":685},[686],{"items":687},[688,693,699,701,706,711,716,721,726,731],{"text":689,"config":690},"About",{"href":691,"dataGaName":692,"dataGaLocation":415},"/company/","about",{"text":694,"config":695,"footerGa":698},"Jobs",{"href":696,"dataGaName":697,"dataGaLocation":415},"/jobs/","jobs",{"dataGaName":697},{"text":660,"config":700},{"href":662,"dataGaName":663,"dataGaLocation":415},{"text":702,"config":703},"Leadership",{"href":704,"dataGaName":705,"dataGaLocation":415},"/company/team/e-group/","leadership",{"text":707,"config":708},"Handbook",{"href":709,"dataGaName":710,"dataGaLocation":415},"https://handbook.gitlab.com/","handbook",{"text":712,"config":713},"Investor relations",{"href":714,"dataGaName":715,"dataGaLocation":415},"https://ir.gitlab.com/overview/default.aspx","investor relations",{"text":717,"config":718},"Trust Center",{"href":719,"dataGaName":720,"dataGaLocation":415},"/security/","trust center",{"text":722,"config":723},"AI Transparency Center",{"href":724,"dataGaName":725,"dataGaLocation":415},"/ai-transparency-center/","ai transparency center",{"text":727,"config":728},"Newsletter",{"href":729,"dataGaName":730,"dataGaLocation":415},"/company/contact/#contact-forms","newsletter",{"text":732,"config":733},"Press",{"href":734,"dataGaName":735,"dataGaLocation":415},"/press/","press",{"text":737,"config":738,"menu":739},"Contact us",{"dataNavLevelOne":683},{"type":466,"columns":740},[741],{"items":742},[743,748,753],{"text":744,"config":745},"Talk to sales",{"href":746,"dataGaName":747,"dataGaLocation":415},"/sales/","talk to sales",{"text":749,"config":750},"Support portal",{"href":751,"dataGaName":752,"dataGaLocation":415},"https://support.gitlab.com/hc/en-us","support portal",{"text":754,"config":755},"Customer portal",{"href":756,"dataGaName":757,"dataGaLocation":415},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":759,"login":760,"suggestions":767},"Close",{"text":761,"link":762},"To search repositories and projects, login to",{"text":763,"config":764},"gitlab.com",{"href":429,"dataGaName":765,"dataGaLocation":766},"search login","search",{"text":768,"default":769},"Suggestions",[770,772,776,778,782,786],{"text":446,"config":771},{"href":451,"dataGaName":446,"dataGaLocation":766},{"text":773,"config":774},"Code Suggestions (AI)",{"href":775,"dataGaName":773,"dataGaLocation":766},"/solutions/code-suggestions/",{"text":482,"config":777},{"href":484,"dataGaName":482,"dataGaLocation":766},{"text":779,"config":780},"GitLab on AWS",{"href":781,"dataGaName":779,"dataGaLocation":766},"/partners/technology-partners/aws/",{"text":783,"config":784},"GitLab on Google Cloud",{"href":785,"dataGaName":783,"dataGaLocation":766},"/partners/technology-partners/google-cloud-platform/",{"text":787,"config":788},"Why GitLab?",{"href":459,"dataGaName":787,"dataGaLocation":766},{"freeTrial":790,"mobileIcon":795,"desktopIcon":800,"secondaryButton":803},{"text":791,"config":792},"Start free trial",{"href":793,"dataGaName":420,"dataGaLocation":794},"https://gitlab.com/-/trials/new/","nav",{"altText":796,"config":797},"Gitlab Icon",{"src":798,"dataGaName":799,"dataGaLocation":794},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":796,"config":801},{"src":802,"dataGaName":799,"dataGaLocation":794},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":804,"config":805},"Get Started",{"href":806,"dataGaName":807,"dataGaLocation":794},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/get-started/","get started",{"freeTrial":809,"mobileIcon":813,"desktopIcon":815},{"text":810,"config":811},"Learn more about GitLab Duo",{"href":451,"dataGaName":812,"dataGaLocation":794},"gitlab duo",{"altText":796,"config":814},{"src":798,"dataGaName":799,"dataGaLocation":794},{"altText":796,"config":816},{"src":802,"dataGaName":799,"dataGaLocation":794},{"button":818,"mobileIcon":823,"desktopIcon":825},{"text":819,"config":820},"/switch",{"href":821,"dataGaName":822,"dataGaLocation":794},"#contact","switch",{"altText":796,"config":824},{"src":798,"dataGaName":799,"dataGaLocation":794},{"altText":796,"config":826},{"src":827,"dataGaName":799,"dataGaLocation":794},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1773335277/ohhpiuoxoldryzrnhfrh.png",{"freeTrial":829,"mobileIcon":834,"desktopIcon":836},{"text":830,"config":831},"Back to pricing",{"href":564,"dataGaName":832,"dataGaLocation":794,"icon":833},"back to pricing","GoBack",{"altText":796,"config":835},{"src":798,"dataGaName":799,"dataGaLocation":794},{"altText":796,"config":837},{"src":802,"dataGaName":799,"dataGaLocation":794},{"title":839,"titleMobile":840,"button":841,"config":846},"Duo Agent Platform delivers 400% ROI, per new Forrester Consulting study.","400% ROI: Forrester TEI for GitLab Duo",{"text":457,"config":842},{"href":843,"dataGaName":844,"dataGaLocation":845},"https://about.gitlab.com/blog/gitlab-duo-agent-platform-delivers-400-percent-roi/","forrester-tei-dap-banner","global-banner",{"layout":847,"disabled":395},"release",{"data":849},{"text":850,"source":851,"edit":857,"contribute":862,"config":867,"items":872,"minimal":1082},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":852,"config":853},"View page source",{"href":854,"dataGaName":855,"dataGaLocation":856},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":858,"config":859},"Edit this page",{"href":860,"dataGaName":861,"dataGaLocation":856},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":863,"config":864},"Please contribute",{"href":865,"dataGaName":866,"dataGaLocation":856},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":868,"facebook":869,"youtube":870,"linkedin":871},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[873,920,974,1018,1050],{"title":562,"links":874,"subMenu":889},[875,879,884],{"text":876,"config":877},"View plans",{"href":564,"dataGaName":878,"dataGaLocation":856},"view plans",{"text":880,"config":881},"Why Premium?",{"href":882,"dataGaName":883,"dataGaLocation":856},"/pricing/premium/","why premium",{"text":885,"config":886},"Why Ultimate?",{"href":887,"dataGaName":888,"dataGaLocation":856},"/pricing/ultimate/","why ultimate",[890],{"title":891,"links":892},"Contact Us",[893,896,898,900,905,910,915],{"text":894,"config":895},"Contact sales",{"href":746,"dataGaName":425,"dataGaLocation":856},{"text":749,"config":897},{"href":751,"dataGaName":752,"dataGaLocation":856},{"text":754,"config":899},{"href":756,"dataGaName":757,"dataGaLocation":856},{"text":901,"config":902},"Status",{"href":903,"dataGaName":904,"dataGaLocation":856},"https://status.gitlab.com/","status",{"text":906,"config":907},"Terms of use",{"href":908,"dataGaName":909,"dataGaLocation":856},"/terms/","terms of use",{"text":911,"config":912},"Privacy statement",{"href":913,"dataGaName":914,"dataGaLocation":856},"/privacy/","privacy statement",{"text":916,"config":917},"Cookie preferences",{"dataGaName":918,"dataGaLocation":856,"id":919,"isOneTrustButton":398},"cookie preferences","ot-sdk-btn",{"title":462,"links":921,"subMenu":930},[922,926],{"text":923,"config":924},"DevSecOps platform",{"href":444,"dataGaName":925,"dataGaLocation":856},"devsecops platform",{"text":927,"config":928},"AI-Assisted Development",{"href":451,"dataGaName":929,"dataGaLocation":856},"ai-assisted development",[931],{"title":932,"links":933},"Topics",[934,939,944,949,954,959,964,969],{"text":935,"config":936},"CICD",{"href":937,"dataGaName":938,"dataGaLocation":856},"/topics/ci-cd/","cicd",{"text":940,"config":941},"GitOps",{"href":942,"dataGaName":943,"dataGaLocation":856},"/topics/gitops/","gitops",{"text":945,"config":946},"DevOps",{"href":947,"dataGaName":948,"dataGaLocation":856},"/topics/devops/","devops",{"text":950,"config":951},"Version Control",{"href":952,"dataGaName":953,"dataGaLocation":856},"/topics/version-control/","version control",{"text":955,"config":956},"DevSecOps",{"href":957,"dataGaName":958,"dataGaLocation":856},"/topics/devsecops/","devsecops",{"text":960,"config":961},"Cloud Native",{"href":962,"dataGaName":963,"dataGaLocation":856},"/topics/cloud-native/","cloud native",{"text":965,"config":966},"AI for Coding",{"href":967,"dataGaName":968,"dataGaLocation":856},"/topics/devops/ai-for-coding/","ai for coding",{"text":970,"config":971},"Agentic AI",{"href":972,"dataGaName":973,"dataGaLocation":856},"/topics/agentic-ai/","agentic ai",{"title":975,"links":976},"Solutions",[977,979,981,986,990,993,997,1000,1002,1005,1008,1013],{"text":506,"config":978},{"href":501,"dataGaName":506,"dataGaLocation":856},{"text":495,"config":980},{"href":478,"dataGaName":479,"dataGaLocation":856},{"text":982,"config":983},"Agile development",{"href":984,"dataGaName":985,"dataGaLocation":856},"/solutions/agile-delivery/","agile delivery",{"text":987,"config":988},"SCM",{"href":491,"dataGaName":989,"dataGaLocation":856},"source code management",{"text":935,"config":991},{"href":484,"dataGaName":992,"dataGaLocation":856},"continuous integration & delivery",{"text":994,"config":995},"Value stream management",{"href":534,"dataGaName":996,"dataGaLocation":856},"value stream management",{"text":940,"config":998},{"href":999,"dataGaName":943,"dataGaLocation":856},"/solutions/gitops/",{"text":544,"config":1001},{"href":547,"dataGaName":548,"dataGaLocation":856},{"text":1003,"config":1004},"Small business",{"href":553,"dataGaName":554,"dataGaLocation":856},{"text":1006,"config":1007},"Public sector",{"href":559,"dataGaName":560,"dataGaLocation":856},{"text":1009,"config":1010},"Education",{"href":1011,"dataGaName":1012,"dataGaLocation":856},"/solutions/education/","education",{"text":1014,"config":1015},"Financial services",{"href":1016,"dataGaName":1017,"dataGaLocation":856},"/solutions/finance/","financial services",{"title":567,"links":1019},[1020,1022,1024,1026,1029,1031,1034,1036,1038,1040,1042,1044,1046,1048],{"text":580,"config":1021},{"href":582,"dataGaName":583,"dataGaLocation":856},{"text":585,"config":1023},{"href":587,"dataGaName":588,"dataGaLocation":856},{"text":590,"config":1025},{"href":592,"dataGaName":593,"dataGaLocation":856},{"text":595,"config":1027},{"href":597,"dataGaName":1028,"dataGaLocation":856},"docs",{"text":618,"config":1030},{"href":620,"dataGaName":621,"dataGaLocation":856},{"text":1032,"config":1033},"What's new",{"href":678,"dataGaName":679,"dataGaLocation":856},{"text":613,"config":1035},{"href":615,"dataGaName":616,"dataGaLocation":856},{"text":632,"config":1037},{"href":634,"dataGaName":635,"dataGaLocation":856},{"text":640,"config":1039},{"href":642,"dataGaName":643,"dataGaLocation":856},{"text":645,"config":1041},{"href":647,"dataGaName":648,"dataGaLocation":856},{"text":650,"config":1043},{"href":652,"dataGaName":653,"dataGaLocation":856},{"text":655,"config":1045},{"href":657,"dataGaName":658,"dataGaLocation":856},{"text":660,"config":1047},{"href":662,"dataGaName":663,"dataGaLocation":856},{"text":665,"config":1049},{"href":667,"dataGaName":668,"dataGaLocation":856},{"title":681,"links":1051},[1052,1054,1056,1058,1060,1062,1066,1071,1073,1075,1077],{"text":689,"config":1053},{"href":691,"dataGaName":683,"dataGaLocation":856},{"text":694,"config":1055},{"href":696,"dataGaName":697,"dataGaLocation":856},{"text":702,"config":1057},{"href":704,"dataGaName":705,"dataGaLocation":856},{"text":707,"config":1059},{"href":709,"dataGaName":710,"dataGaLocation":856},{"text":712,"config":1061},{"href":714,"dataGaName":715,"dataGaLocation":856},{"text":1063,"config":1064},"Sustainability",{"href":1065,"dataGaName":1063,"dataGaLocation":856},"/sustainability/",{"text":1067,"config":1068},"Diversity, inclusion and belonging (DIB)",{"href":1069,"dataGaName":1070,"dataGaLocation":856},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":717,"config":1072},{"href":719,"dataGaName":720,"dataGaLocation":856},{"text":727,"config":1074},{"href":729,"dataGaName":730,"dataGaLocation":856},{"text":732,"config":1076},{"href":734,"dataGaName":735,"dataGaLocation":856},{"text":1078,"config":1079},"Modern Slavery Transparency Statement",{"href":1080,"dataGaName":1081,"dataGaLocation":856},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"items":1083},[1084,1087,1090],{"text":1085,"config":1086},"Terms",{"href":908,"dataGaName":909,"dataGaLocation":856},{"text":1088,"config":1089},"Cookies",{"dataGaName":918,"dataGaLocation":856,"id":919,"isOneTrustButton":398},{"text":1091,"config":1092},"Privacy",{"href":913,"dataGaName":914,"dataGaLocation":856},[1094],{"id":1095,"title":7,"body":394,"config":1096,"content":1098,"description":394,"extension":1101,"meta":1102,"navigation":398,"path":1103,"seo":1104,"stem":1105,"__hash__":1106},"blogAuthors/en-us/blog/authors/steve-azzopardi.yml",{"template":1097},"BlogAuthor",{"name":7,"config":1099},{"headshot":160,"ctfId":1100},"steveazz","yml",{},"/en-us/blog/authors/steve-azzopardi",{},"en-us/blog/authors/steve-azzopardi","qnb6MwMS78l86ZK9JZJ-5qc-NcHCq2bdzgXn2tHRN40",[1108,1117,1125],{"title":1109,"description":1110,"heroImage":1111,"category":390,"date":1112,"authors":1113,"slug":1116,"externalUrl":394},"Confidential AI for GitLab Self-Hosted","Give developers AI coding agents in GitLab Duo without source code leaving a hardware-encrypted boundary — no GPUs needed.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1773866173/vte9qh8rriznvyclhkes.png","2026-08-06",[1114,1115],"Mathias Ewald","Martin Paloncy, Edgeless Systems","confidential-ai-for-gitlab-self-hosted",{"title":1118,"description":1119,"heroImage":1120,"category":390,"date":1121,"authors":1122,"slug":1124,"externalUrl":394},"Green DevOps: Why carbon measurement belongs in your CI/CD pipeline","CI/CD pipelines have a hidden carbon cost. Here's why measuring it matters, and how you can get started with Eco CI and Carmen in GitLab.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1765809212/noh0mdfn9o94ry9ykura.png","2026-07-09",[1123],"Lysanne Pinto","green-devops-carbon-measurement-cicd-pipeline",{"title":1126,"description":1127,"heroImage":1128,"category":390,"date":1129,"authors":1130,"slug":1132,"externalUrl":394},"How to build CI/CD observability at scale","This practical guide to GitLab pipeline analytics helps self-managed users gain operational insights using Prometheus and Grafana.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1774465167/n5hlvrsrheadeccyr1oz.png","2026-04-28",[1131],"Paul Meresanu","how-to-build-ci-cd-observability-at-scale",{"promotions":1134},[1135,1149,1161,1173],{"id":1136,"categories":1137,"header":1139,"text":1140,"button":1141,"image":1146},"ai-modernization",[1138],"ai","Is AI achieving its promise at scale?","Quiz will take 5 minutes or less",{"text":1142,"config":1143},"Get your AI maturity score",{"href":1144,"dataGaName":1145,"dataGaLocation":621},"/assessments/ai-modernization-assessment/","modernization assessment",{"config":1147},{"src":1148},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138786/qix0m7kwnd8x2fh1zq49.png",{"id":1150,"categories":1151,"header":1153,"text":1140,"button":1154,"image":1158},"devops-modernization",[1152,958],"product","Are you just managing tools or shipping innovation?",{"text":1155,"config":1156},"Get your DevOps maturity score",{"href":1157,"dataGaName":1145,"dataGaLocation":621},"/assessments/devops-modernization-assessment/",{"config":1159},{"src":1160},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138785/eg818fmakweyuznttgid.png",{"id":1162,"categories":1163,"header":1165,"text":1140,"button":1166,"image":1170},"security-modernization",[1164],"security","Are you trading speed for security?",{"text":1167,"config":1168},"Get your security maturity score",{"href":1169,"dataGaName":1145,"dataGaLocation":621},"/assessments/security-modernization-assessment/",{"config":1171},{"src":1172},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138786/p4pbqd9nnjejg5ds6mdk.png",{"id":1174,"paths":1175,"header":1178,"text":1179,"button":1180,"image":1185},"github-azure-migration",[1176,1177],"migration-from-azure-devops-to-gitlab","integrating-azure-devops-scm-and-gitlab","Is your team ready for GitHub's Azure move?","GitHub is already rebuilding around Azure. Find out what it means for you.",{"text":1181,"config":1182},"See how GitLab compares to GitHub",{"href":1183,"dataGaName":1184,"dataGaLocation":621},"/compare/gitlab-vs-github/github-azure-migration/","github azure migration",{"config":1186},{"src":1160},{"header":1188,"blurb":1189,"button":1190,"secondaryButton":1195},"Start building faster today","See what your team can do with the intelligent orchestration platform for DevSecOps.\n",{"text":1191,"config":1192},"Get your free trial",{"href":1193,"dataGaName":420,"dataGaLocation":1194},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":894,"config":1196},{"href":746,"dataGaName":425,"dataGaLocation":1194},1786803737173]