[{"data":1,"prerenderedAt":1053},["ShallowReactive",2],{"/blog/path-to-decomposing-gitlab-database-part2":3,"navigation-en-us":265,"banner-en-us":693,"footer-en-us":703,"blog-post-authors-en-us-Dylan Griffith":948,"blog-related-posts-en-us-path-to-decomposing-gitlab-database-part2":963,"blog-promotions-en-us":989,"next-steps-en-us":1043},{"id":4,"title":5,"authors":6,"body":8,"category":243,"date":244,"description":245,"extension":246,"externalUrl":247,"faq":247,"featured":248,"heroImage":249,"meta":250,"navigation":251,"path":252,"seo":253,"slug":258,"stem":259,"tags":260,"template":263,"updatedDate":247,"__hash__":264},"blogPosts/en-us/blog/path-to-decomposing-gitlab-database-part2.md","Decomposing the GitLab backend database, Part 2: Final migration and results",[7],"Dylan Griffith",{"type":9,"value":10,"toc":234},"minimark",[11,24,29,38,69,76,79,82,93,108,119,122,126,129,132,135,144,148,151,154,168,172,180,183,223,227],[12,13,14],"p",{},[15,16,17,18,23],"em",{},"This blog post is part 2 in a three-part series about decomposing the GitLab backend database. It focuses on the final migration\nprocess and highlights the results we achieved after the migration. If you want to read about the design and planning phase, check out ",[19,20,22],"a",{"href":21},"/blog/path-to-decomposing-gitlab-database-part1/","part 1",".",[25,26,28],"h2",{"id":27},"deciding-between-zero-downtime-and-full-downtime","Deciding between zero downtime and full downtime",[12,30,31,32,37],{},"Early on in the project we thought it would be necessary for the migration to\nbe \"zero downtime\" or \"near-zero downtime\". We ",[19,33,36],{"href":34,"rel":35},"https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/84588",[],"came up with this plan","\nearly on which involved (in summary):",[39,40,41,45,48,51,54,57,60,63,66],"ol",{},[42,43,44],"li",{},"The entire database would be replicated (including non-CI tables) using\nPatroni cascading/standby replication to a dedicated CI Patroni cluster.\nReplication only lags by at most a few seconds.",[42,46,47],{},"Read traffic for CI tables could be split ahead of time to read from the CI\nreplicas.",[42,49,50],{},"Write traffic would be split ahead of the migration into CI and Main by\nsending these through separate dedicated PGBouncer proxies. Initially CI\nwrites still go to the Main database since the CI cluster is just a standby.\nThese proxies would be the thing we reconfigured during the live migration\nto point at the CI cluster.",[42,52,53],{},"At the time of migration we would pause writes to the CI tables by pausing\nthe CI PGBouncer.",[42,55,56],{},"After pausing writes to the CI database we'd capture the current LSN\nposition in Postgres of the Main primary database (now expect no more writes\nto CI tables to be possible).",[42,58,59],{},"After that we wait until the CI database replication catches up to that\npoint.",[42,61,62],{},"Then we promote the CI database to accept writes (remove the cascading\nreplication).",[42,64,65],{},"Then we reconfigure writes to point to the CI database by updating the write\nhost in the CI PGBouncer.",[42,67,68],{},"The migration is done.",[12,70,71],{},[72,73],"img",{"alt":74,"src":75},"Database architecture actual final migration step","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782397716/blog/Content%20Images/2022-07-15-path-to-decomposing-gitlab-database/phase4to7.png",[12,77,78],{},"This approach (assuming that the CI replicas were only delayed by a few\nseconds) would mean that, at most, there would be a few seconds where CI writes\nmight result in errors and 500s for users. Many failures would likely already\nbe retried since much of CI write traffic goes via asynchronous (Sidekiq)\nprocesses that automatically retry.",[12,80,81],{},"In the end we didn't use this approach because:",[39,83,84,87,90],{},[42,85,86],{},"This approach didn't have an easy-to-implement rollback strategy. Data that\nwas written to CI tables during the migration would be lost if we rolled\nback to just the Main database.",[42,88,89],{},"The period of a few seconds where we expect to see some errors might make it\ndifficult for us to quickly determine the success or failure of the\nmigration.",[42,91,92],{},"There was no hard business requirement to avoid downtime.",[12,94,95,96,101,102,107],{},"The ",[19,97,100],{"href":98,"rel":99},"https://gitlab.com/groups/gitlab-org/-/epics/7791",[],"migration approach we ended up using"," took two\nhours of downtime. We stopped all GitLab services that could read or write\nfrom the database. We also blocked user-level traffic at the CDN (Cloudflare) to allow us\nto do some automated and manual testing before opening traffic back up to\nusers. This allowed us to prepare a ",[19,103,106],{"href":104,"rel":105},"https://gitlab.com/gitlab-org/gitlab/-/issues/361759",[],"slightly more straightforward rollback procedure",",\nwhich was:",[39,109,110,113,116],{},[42,111,112],{},"Reconfigure all read-only CI traffic back to the Main replicas",[42,114,115],{},"Reconfigure all read-write CI traffic (via PGBouncer) back to the Main\nprimary database",[42,117,118],{},"Increment the Postgres sequences for all CI tables to avoid overlapping with\ndata we created in our testing",[12,120,121],{},"Ultimately having a simple rollback mechanism proved very useful in doing many\npractice runs on staging.",[25,123,125],{"id":124},"rehearsing-the-migration-process","Rehearsing the migration process",[12,127,128],{},"Before executing the final migration on GitLab.com, we executed seven rehearsals\nwith rollback and one final migration on our staging environment. In these\npractice runs, we discovered many small issues that would have likely caused\nissues in the production environment.",[12,130,131],{},"These rehearsals also gave all the participants an opportunity to perfect their steps\nin the process to minimize delays in our production rollout. This practice\nultimately allowed us to be quite confident in our timeline of at most two hours of downtime.",[12,133,134],{},"In the end, we finished the migration in 93 minutes, with a few small delays caused by\nsurprises we did not see in staging.",[12,136,137,138,143],{},"The rehearsal process was very time-consuming and a vast effort to execute in\nthe context of GitLab, where we all ",[19,139,142],{"href":140,"rel":141},"https://handbook.gitlab.com/handbook/company/culture/all-remote/asynchronous/",[],"work\nasynchronously","\nand across different timezones. However, it proved to be essential to the success of\nthis project.",[25,145,147],{"id":146},"preparing-for-production-migration","Preparing for production migration",[12,149,150],{},"One week before our the final migration on production we prepared a production\nreadiness review issue for final approval from executives. This was a good\nopportunity to highlight all the preparation and validation we'd done to give\nus confidence in the plan. This also encouraged us to do extra validation where\nwe might expect to see questions or concerns about the plan.",[12,152,153],{},"Some highlights from this review included:",[39,155,156,159,162,165],{},[42,157,158],{},"The amount of practice runs we'd done including details about the problems\nwe'd seen and resolved in staging",[42,160,161],{},"Metrics which we'd observed to prove all the queries were using the right\ndatabase connections already",[42,163,164],{},"Details about how long we'd been running without issues in local development\nwith all GitLab developers running with two databases by default",[42,166,167],{},"Details about the rollback strategy we would use if necessary and how we\ntested this rollback strategy in staging as well as some production\nvalidation",[25,169,171],{"id":170},"tracking-the-results","Tracking the results",[12,173,174,175,23],{},"After we completed the rollout we tracked\n",[19,176,179],{"href":177,"rel":178},"https://gitlab.com/gl-retrospectives/sharding-group/-/issues/18",[],"performance improvements across some metrics we expected to improve",[12,181,182],{},"The data showed:",[184,185,186,196,199,202,205,214],"ul",{},[42,187,188,189,192],{},"We decreased the CPU utilization of our primary database server, giving us much more headroom.",[190,191],"br",{},[72,193],{"alt":194,"src":195},"CPU peaks before and after decomposition shows smaller peaks after","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782397714/blog/Content%20Images/2022-07-15-path-to-decomposing-gitlab-database/cpu-post-decomposition-improvement.png",[42,197,198],{},"We can free around 9.2TiB out of 22TiB from our Main database by truncating the CI tables.",[42,200,201],{},"We can free around 12.5TiB out of 22TiB from our CI database by truncating the Main tables.",[42,203,204],{},"We significantly reduced the rate of dead tuples on our Main database.",[42,206,207,208,210],{},"We significantly reduced vacuuming saturation. Before decomposition the Main database\nmaximum vacuuming saturation was up to 100%, with the average closer to 80%. After\ndecomposition, vacuuming saturation has stabilized at around 15% for\nboth databases.",[190,209],{},[72,211],{"alt":212,"src":213},"Vacuum saturation before and after decomposition shows a decrease after decomposition","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782397716/blog/Content%20Images/2022-07-15-path-to-decomposing-gitlab-database/vacuum-saturation-post-decomposition.png",[42,215,216,217,219],{},"We reduced the average query duration for our Sidekiq PGBouncer query\npool by at least a factor of 5 once we scaled up connection limits due to our\nincreased headroom. Previously we needed to throttle connections for\nasynchronous workloads to avoid overloading the primary database.",[190,218],{},[72,220],{"alt":221,"src":222},"Average active query duration by workload shows a decrease after scaling connections after decomposition","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782397714/blog/Content%20Images/2022-07-15-path-to-decomposing-gitlab-database/pgbouncer-active-query-duration-by-workload.png",[25,224,226],{"id":225},"continue-reading","Continue reading",[12,228,229,230,23],{},"You can read more about some interesting technical challenges and surprises we\nhad to deal with along the way in\n",[19,231,233],{"href":232},"/blog/path-to-decomposing-gitlab-database-part3/","part 3",{"title":235,"searchDepth":236,"depth":236,"links":237},"",2,[238,239,240,241,242],{"id":27,"depth":236,"text":28},{"id":124,"depth":236,"text":125},{"id":146,"depth":236,"text":147},{"id":170,"depth":236,"text":171},{"id":225,"depth":236,"text":226},"engineering","2022-08-04","This is the second in our three-part technical summary of the yearlong project to decompose GitLab's Postgres database.","md",null,false,"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749663397/Blog/Hero%20Images/logoforblogpost.jpg",{},true,"/en-us/blog/path-to-decomposing-gitlab-database-part2",{"title":254,"description":245,"ogTitle":254,"ogDescription":245,"noIndex":248,"ogImage":249,"ogUrl":255,"ogSiteName":256,"ogType":257,"canonicalUrls":255},"GitLab database decomposition: Final migration and results","https://about.gitlab.com/blog/path-to-decomposing-gitlab-database-part2","https://about.gitlab.com","article","path-to-decomposing-gitlab-database-part2","en-us/blog/path-to-decomposing-gitlab-database-part2",[261,262],"inside GitLab","design","BlogPost","nGJq4ZvnpAP6z973exgTjvopQHZCNnl9RNmmW5Thz6M",{"logo":266,"freeTrial":271,"sales":276,"login":281,"items":286,"search":613,"minimal":644,"duo":663,"switchNav":672,"pricingDeployment":683},{"config":267},{"href":268,"dataGaName":269,"dataGaLocation":270},"/","gitlab logo","header",{"text":272,"config":273},"Get free trial",{"href":274,"dataGaName":275,"dataGaLocation":270},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":277,"config":278},"Request a demo",{"href":279,"dataGaName":280,"dataGaLocation":270},"/sales/?contact-topic=request-demo","sales",{"text":282,"config":283},"Sign in",{"href":284,"dataGaName":285,"dataGaLocation":270},"https://gitlab.com/users/sign_in/","sign in",[287,316,416,421,535,591],{"text":288,"config":289,"menu":291},"Platform",{"dataNavLevelOne":290},"platform",{"type":292,"columns":293},"cards",[294,300,308],{"title":288,"description":295,"link":296},"The intelligent orchestration platform for DevSecOps",{"text":297,"config":298},"Explore our Platform",{"href":299,"dataGaName":290,"dataGaLocation":270},"/platform/",{"title":301,"description":302,"link":303},"GitLab Duo Agent Platform","Agentic AI for the entire software lifecycle",{"text":304,"config":305},"Meet GitLab Duo",{"href":306,"dataGaName":307,"dataGaLocation":270},"/gitlab-duo-agent-platform/","gitlab duo agent platform",{"title":309,"description":310,"link":311},"Why GitLab","See the top reasons enterprises choose GitLab",{"text":312,"config":313},"Learn more",{"href":314,"dataGaName":315,"dataGaLocation":270},"/why-gitlab/","why gitlab",{"text":317,"left":251,"config":318,"menu":320},"Product",{"dataNavLevelOne":319},"solutions",{"type":321,"link":322,"columns":326,"feature":395},"lists",{"text":323,"config":324},"View all Solutions",{"href":325,"dataGaName":319,"dataGaLocation":270},"/solutions/",[327,351,374],{"title":328,"description":329,"link":330,"items":335},"Automation","CI/CD and automation to accelerate deployment",{"config":331},{"icon":332,"href":333,"dataGaName":334,"dataGaLocation":270},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[336,340,343,347],{"text":337,"config":338},"CI/CD",{"href":339,"dataGaLocation":270,"dataGaName":337},"/solutions/continuous-integration/",{"text":301,"config":341},{"href":306,"dataGaLocation":270,"dataGaName":342},"gitlab duo agent platform - product menu",{"text":344,"config":345},"Source Code Management",{"href":346,"dataGaLocation":270,"dataGaName":344},"/solutions/source-code-management/",{"text":348,"config":349},"Automated Software Delivery",{"href":333,"dataGaLocation":270,"dataGaName":350},"Automated software delivery",{"title":352,"description":353,"link":354,"items":359},"Security","Deliver code faster without compromising security",{"config":355},{"href":356,"dataGaName":357,"dataGaLocation":270,"icon":358},"/solutions/application-security-testing/","security and compliance","ShieldCheckLight",[360,364,369],{"text":361,"config":362},"Application Security Testing",{"href":356,"dataGaName":363,"dataGaLocation":270},"Application security testing",{"text":365,"config":366},"Software Supply Chain Security",{"href":367,"dataGaLocation":270,"dataGaName":368},"/solutions/supply-chain/","Software supply chain security",{"text":370,"config":371},"Software Compliance",{"href":372,"dataGaName":373,"dataGaLocation":270},"/solutions/software-compliance/","software compliance",{"title":375,"link":376,"items":381},"Measurement",{"config":377},{"icon":378,"href":379,"dataGaName":380,"dataGaLocation":270},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[382,386,390],{"text":383,"config":384},"Visibility & Measurement",{"href":379,"dataGaLocation":270,"dataGaName":385},"Visibility and Measurement",{"text":387,"config":388},"Value Stream Management",{"href":389,"dataGaLocation":270,"dataGaName":387},"/solutions/value-stream-management/",{"text":391,"config":392},"Analytics & Insights",{"href":393,"dataGaLocation":270,"dataGaName":394},"/solutions/analytics-and-insights/","Analytics and insights",{"title":396,"type":321,"items":397},"GitLab for",[398,404,410],{"text":399,"config":400},"Enterprise",{"icon":401,"href":402,"dataGaLocation":270,"dataGaName":403},"Building","/enterprise/","enterprise",{"text":405,"config":406},"Small Business",{"icon":407,"href":408,"dataGaLocation":270,"dataGaName":409},"Work","/small-business/","small business",{"text":411,"config":412},"Public Sector",{"icon":413,"href":414,"dataGaLocation":270,"dataGaName":415},"Organization","/solutions/public-sector/","public sector",{"text":417,"config":418},"Pricing",{"href":419,"dataGaName":420,"dataGaLocation":270,"dataNavLevelOne":420},"/pricing/","pricing",{"text":422,"config":423,"menu":425},"Resources",{"dataNavLevelOne":424},"resources",{"type":321,"link":426,"columns":430,"feature":524},{"text":427,"config":428},"View all resources",{"href":429,"dataGaName":424,"dataGaLocation":270},"/resources/",[431,464,491],{"title":432,"items":433},"Getting started",[434,439,444,449,454,459],{"text":435,"config":436},"Install",{"href":437,"dataGaName":438,"dataGaLocation":270},"/install/","install",{"text":440,"config":441},"Quick start guides",{"href":442,"dataGaName":443,"dataGaLocation":270},"/get-started/","quick setup checklists",{"text":445,"config":446},"Learn",{"href":447,"dataGaLocation":270,"dataGaName":448},"https://university.gitlab.com/","learn",{"text":450,"config":451},"Product documentation",{"href":452,"dataGaName":453,"dataGaLocation":270},"https://docs.gitlab.com/","product documentation",{"text":455,"config":456},"Best practice videos",{"href":457,"dataGaName":458,"dataGaLocation":270},"/getting-started-videos/","best practice videos",{"text":460,"config":461},"Integrations",{"href":462,"dataGaName":463,"dataGaLocation":270},"/integrations/","integrations",{"title":465,"items":466},"Discover",[467,472,477,482,486],{"text":468,"config":469},"Customer success stories",{"href":470,"dataGaName":471,"dataGaLocation":270},"/customers/","customer success stories",{"text":473,"config":474},"Blog",{"href":475,"dataGaName":476,"dataGaLocation":270},"/blog/","blog",{"text":478,"config":479},"Demo Hub",{"href":480,"dataGaName":481,"dataGaLocation":270},"/demo-hub/","demo hub",{"text":483,"config":484},"The Source",{"href":485,"dataGaName":476,"dataGaLocation":270},"/the-source/",{"text":487,"config":488},"Remote",{"href":489,"dataGaName":490,"dataGaLocation":270},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"title":492,"items":493},"Connect",[494,499,504,509,514,519],{"text":495,"config":496},"GitLab Services",{"href":497,"dataGaName":498,"dataGaLocation":270},"/services/","services",{"text":500,"config":501},"Contribute",{"href":502,"dataGaName":503,"dataGaLocation":270},"https://contributors.gitlab.com","contribute",{"text":505,"config":506},"Community",{"href":507,"dataGaName":508,"dataGaLocation":270},"/community/","community",{"text":510,"config":511},"Forum",{"href":512,"dataGaName":513,"dataGaLocation":270},"https://forum.gitlab.com/","forum",{"text":515,"config":516},"Events",{"href":517,"dataGaName":518,"dataGaLocation":270},"/events/","events",{"text":520,"config":521},"Partners",{"href":522,"dataGaName":523,"dataGaLocation":270},"/partners/","partners",{"config":525,"title":528,"text":529,"link":530},{"background":526,"textColor":527},"url('https://res.cloudinary.com/about-gitlab-com/image/upload/v1777322348/qpq8yrgn8knii57omj0c.png')","#000","What’s new in GitLab","Stay updated with our latest features and improvements.",{"text":531,"config":532},"Read the latest",{"href":533,"dataGaName":534,"dataGaLocation":270},"/whats-new/","whats new",{"text":536,"config":537,"menu":539},"Company",{"dataNavLevelOne":538},"company",{"type":321,"columns":540},[541],{"items":542},[543,548,554,556,561,566,571,576,581,586],{"text":544,"config":545},"About",{"href":546,"dataGaName":547,"dataGaLocation":270},"/company/","about",{"text":549,"config":550,"footerGa":553},"Jobs",{"href":551,"dataGaName":552,"dataGaLocation":270},"/jobs/","jobs",{"dataGaName":552},{"text":515,"config":555},{"href":517,"dataGaName":518,"dataGaLocation":270},{"text":557,"config":558},"Leadership",{"href":559,"dataGaName":560,"dataGaLocation":270},"/company/team/e-group/","leadership",{"text":562,"config":563},"Handbook",{"href":564,"dataGaName":565,"dataGaLocation":270},"https://handbook.gitlab.com/","handbook",{"text":567,"config":568},"Investor relations",{"href":569,"dataGaName":570,"dataGaLocation":270},"https://ir.gitlab.com/overview/default.aspx","investor relations",{"text":572,"config":573},"Trust Center",{"href":574,"dataGaName":575,"dataGaLocation":270},"/security/","trust center",{"text":577,"config":578},"AI Transparency Center",{"href":579,"dataGaName":580,"dataGaLocation":270},"/ai-transparency-center/","ai transparency center",{"text":582,"config":583},"Newsletter",{"href":584,"dataGaName":585,"dataGaLocation":270},"/company/contact/#contact-forms","newsletter",{"text":587,"config":588},"Press",{"href":589,"dataGaName":590,"dataGaLocation":270},"/press/","press",{"text":592,"config":593,"menu":594},"Contact us",{"dataNavLevelOne":538},{"type":321,"columns":595},[596],{"items":597},[598,603,608],{"text":599,"config":600},"Talk to sales",{"href":601,"dataGaName":602,"dataGaLocation":270},"/sales/","talk to sales",{"text":604,"config":605},"Support portal",{"href":606,"dataGaName":607,"dataGaLocation":270},"https://support.gitlab.com/hc/en-us","support portal",{"text":609,"config":610},"Customer portal",{"href":611,"dataGaName":612,"dataGaLocation":270},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":614,"login":615,"suggestions":622},"Close",{"text":616,"link":617},"To search repositories and projects, login to",{"text":618,"config":619},"gitlab.com",{"href":284,"dataGaName":620,"dataGaLocation":621},"search login","search",{"text":623,"default":624},"Suggestions",[625,627,631,633,637,641],{"text":301,"config":626},{"href":306,"dataGaName":301,"dataGaLocation":621},{"text":628,"config":629},"Code Suggestions (AI)",{"href":630,"dataGaName":628,"dataGaLocation":621},"/solutions/code-suggestions/",{"text":337,"config":632},{"href":339,"dataGaName":337,"dataGaLocation":621},{"text":634,"config":635},"GitLab on AWS",{"href":636,"dataGaName":634,"dataGaLocation":621},"/partners/technology-partners/aws/",{"text":638,"config":639},"GitLab on Google Cloud",{"href":640,"dataGaName":638,"dataGaLocation":621},"/partners/technology-partners/google-cloud-platform/",{"text":642,"config":643},"Why GitLab?",{"href":314,"dataGaName":642,"dataGaLocation":621},{"freeTrial":645,"mobileIcon":650,"desktopIcon":655,"secondaryButton":658},{"text":646,"config":647},"Start free trial",{"href":648,"dataGaName":275,"dataGaLocation":649},"https://gitlab.com/-/trials/new/","nav",{"altText":651,"config":652},"Gitlab Icon",{"src":653,"dataGaName":654,"dataGaLocation":649},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":651,"config":656},{"src":657,"dataGaName":654,"dataGaLocation":649},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":659,"config":660},"Get Started",{"href":661,"dataGaName":662,"dataGaLocation":649},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/get-started/","get started",{"freeTrial":664,"mobileIcon":668,"desktopIcon":670},{"text":665,"config":666},"Learn more about GitLab Duo",{"href":306,"dataGaName":667,"dataGaLocation":649},"gitlab duo",{"altText":651,"config":669},{"src":653,"dataGaName":654,"dataGaLocation":649},{"altText":651,"config":671},{"src":657,"dataGaName":654,"dataGaLocation":649},{"button":673,"mobileIcon":678,"desktopIcon":680},{"text":674,"config":675},"/switch",{"href":676,"dataGaName":677,"dataGaLocation":649},"#contact","switch",{"altText":651,"config":679},{"src":653,"dataGaName":654,"dataGaLocation":649},{"altText":651,"config":681},{"src":682,"dataGaName":654,"dataGaLocation":649},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1773335277/ohhpiuoxoldryzrnhfrh.png",{"freeTrial":684,"mobileIcon":689,"desktopIcon":691},{"text":685,"config":686},"Back to pricing",{"href":419,"dataGaName":687,"dataGaLocation":649,"icon":688},"back to pricing","GoBack",{"altText":651,"config":690},{"src":653,"dataGaName":654,"dataGaLocation":649},{"altText":651,"config":692},{"src":657,"dataGaName":654,"dataGaLocation":649},{"title":694,"titleMobile":695,"button":696,"config":701},"Duo Agent Platform delivers 400% ROI, per new Forrester Consulting study.","400% ROI: Forrester TEI for GitLab Duo",{"text":312,"config":697},{"href":698,"dataGaName":699,"dataGaLocation":700},"https://about.gitlab.com/blog/gitlab-duo-agent-platform-delivers-400-percent-roi/","forrester-tei-dap-banner","global-banner",{"layout":702,"disabled":248},"release",{"data":704},{"text":705,"source":706,"edit":712,"contribute":717,"config":722,"items":727,"minimal":937},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":707,"config":708},"View page source",{"href":709,"dataGaName":710,"dataGaLocation":711},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":713,"config":714},"Edit this page",{"href":715,"dataGaName":716,"dataGaLocation":711},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":718,"config":719},"Please contribute",{"href":720,"dataGaName":721,"dataGaLocation":711},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":723,"facebook":724,"youtube":725,"linkedin":726},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[728,775,829,873,905],{"title":417,"links":729,"subMenu":744},[730,734,739],{"text":731,"config":732},"View plans",{"href":419,"dataGaName":733,"dataGaLocation":711},"view plans",{"text":735,"config":736},"Why Premium?",{"href":737,"dataGaName":738,"dataGaLocation":711},"/pricing/premium/","why premium",{"text":740,"config":741},"Why Ultimate?",{"href":742,"dataGaName":743,"dataGaLocation":711},"/pricing/ultimate/","why ultimate",[745],{"title":746,"links":747},"Contact Us",[748,751,753,755,760,765,770],{"text":749,"config":750},"Contact sales",{"href":601,"dataGaName":280,"dataGaLocation":711},{"text":604,"config":752},{"href":606,"dataGaName":607,"dataGaLocation":711},{"text":609,"config":754},{"href":611,"dataGaName":612,"dataGaLocation":711},{"text":756,"config":757},"Status",{"href":758,"dataGaName":759,"dataGaLocation":711},"https://status.gitlab.com/","status",{"text":761,"config":762},"Terms of use",{"href":763,"dataGaName":764,"dataGaLocation":711},"/terms/","terms of use",{"text":766,"config":767},"Privacy statement",{"href":768,"dataGaName":769,"dataGaLocation":711},"/privacy/","privacy statement",{"text":771,"config":772},"Cookie preferences",{"dataGaName":773,"dataGaLocation":711,"id":774,"isOneTrustButton":251},"cookie preferences","ot-sdk-btn",{"title":317,"links":776,"subMenu":785},[777,781],{"text":778,"config":779},"DevSecOps platform",{"href":299,"dataGaName":780,"dataGaLocation":711},"devsecops platform",{"text":782,"config":783},"AI-Assisted Development",{"href":306,"dataGaName":784,"dataGaLocation":711},"ai-assisted development",[786],{"title":787,"links":788},"Topics",[789,794,799,804,809,814,819,824],{"text":790,"config":791},"CICD",{"href":792,"dataGaName":793,"dataGaLocation":711},"/topics/ci-cd/","cicd",{"text":795,"config":796},"GitOps",{"href":797,"dataGaName":798,"dataGaLocation":711},"/topics/gitops/","gitops",{"text":800,"config":801},"DevOps",{"href":802,"dataGaName":803,"dataGaLocation":711},"/topics/devops/","devops",{"text":805,"config":806},"Version Control",{"href":807,"dataGaName":808,"dataGaLocation":711},"/topics/version-control/","version control",{"text":810,"config":811},"DevSecOps",{"href":812,"dataGaName":813,"dataGaLocation":711},"/topics/devsecops/","devsecops",{"text":815,"config":816},"Cloud Native",{"href":817,"dataGaName":818,"dataGaLocation":711},"/topics/cloud-native/","cloud native",{"text":820,"config":821},"AI for Coding",{"href":822,"dataGaName":823,"dataGaLocation":711},"/topics/devops/ai-for-coding/","ai for coding",{"text":825,"config":826},"Agentic AI",{"href":827,"dataGaName":828,"dataGaLocation":711},"/topics/agentic-ai/","agentic ai",{"title":830,"links":831},"Solutions",[832,834,836,841,845,848,852,855,857,860,863,868],{"text":361,"config":833},{"href":356,"dataGaName":361,"dataGaLocation":711},{"text":350,"config":835},{"href":333,"dataGaName":334,"dataGaLocation":711},{"text":837,"config":838},"Agile development",{"href":839,"dataGaName":840,"dataGaLocation":711},"/solutions/agile-delivery/","agile delivery",{"text":842,"config":843},"SCM",{"href":346,"dataGaName":844,"dataGaLocation":711},"source code management",{"text":790,"config":846},{"href":339,"dataGaName":847,"dataGaLocation":711},"continuous integration & delivery",{"text":849,"config":850},"Value stream management",{"href":389,"dataGaName":851,"dataGaLocation":711},"value stream management",{"text":795,"config":853},{"href":854,"dataGaName":798,"dataGaLocation":711},"/solutions/gitops/",{"text":399,"config":856},{"href":402,"dataGaName":403,"dataGaLocation":711},{"text":858,"config":859},"Small business",{"href":408,"dataGaName":409,"dataGaLocation":711},{"text":861,"config":862},"Public sector",{"href":414,"dataGaName":415,"dataGaLocation":711},{"text":864,"config":865},"Education",{"href":866,"dataGaName":867,"dataGaLocation":711},"/solutions/education/","education",{"text":869,"config":870},"Financial services",{"href":871,"dataGaName":872,"dataGaLocation":711},"/solutions/finance/","financial services",{"title":422,"links":874},[875,877,879,881,884,886,889,891,893,895,897,899,901,903],{"text":435,"config":876},{"href":437,"dataGaName":438,"dataGaLocation":711},{"text":440,"config":878},{"href":442,"dataGaName":443,"dataGaLocation":711},{"text":445,"config":880},{"href":447,"dataGaName":448,"dataGaLocation":711},{"text":450,"config":882},{"href":452,"dataGaName":883,"dataGaLocation":711},"docs",{"text":473,"config":885},{"href":475,"dataGaName":476,"dataGaLocation":711},{"text":887,"config":888},"What's new",{"href":533,"dataGaName":534,"dataGaLocation":711},{"text":468,"config":890},{"href":470,"dataGaName":471,"dataGaLocation":711},{"text":487,"config":892},{"href":489,"dataGaName":490,"dataGaLocation":711},{"text":495,"config":894},{"href":497,"dataGaName":498,"dataGaLocation":711},{"text":500,"config":896},{"href":502,"dataGaName":503,"dataGaLocation":711},{"text":505,"config":898},{"href":507,"dataGaName":508,"dataGaLocation":711},{"text":510,"config":900},{"href":512,"dataGaName":513,"dataGaLocation":711},{"text":515,"config":902},{"href":517,"dataGaName":518,"dataGaLocation":711},{"text":520,"config":904},{"href":522,"dataGaName":523,"dataGaLocation":711},{"title":536,"links":906},[907,909,911,913,915,917,921,926,928,930,932],{"text":544,"config":908},{"href":546,"dataGaName":538,"dataGaLocation":711},{"text":549,"config":910},{"href":551,"dataGaName":552,"dataGaLocation":711},{"text":557,"config":912},{"href":559,"dataGaName":560,"dataGaLocation":711},{"text":562,"config":914},{"href":564,"dataGaName":565,"dataGaLocation":711},{"text":567,"config":916},{"href":569,"dataGaName":570,"dataGaLocation":711},{"text":918,"config":919},"Sustainability",{"href":920,"dataGaName":918,"dataGaLocation":711},"/sustainability/",{"text":922,"config":923},"Diversity, inclusion and belonging (DIB)",{"href":924,"dataGaName":925,"dataGaLocation":711},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":572,"config":927},{"href":574,"dataGaName":575,"dataGaLocation":711},{"text":582,"config":929},{"href":584,"dataGaName":585,"dataGaLocation":711},{"text":587,"config":931},{"href":589,"dataGaName":590,"dataGaLocation":711},{"text":933,"config":934},"Modern Slavery Transparency Statement",{"href":935,"dataGaName":936,"dataGaLocation":711},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"items":938},[939,942,945],{"text":940,"config":941},"Terms",{"href":763,"dataGaName":764,"dataGaLocation":711},{"text":943,"config":944},"Cookies",{"dataGaName":773,"dataGaLocation":711,"id":774,"isOneTrustButton":251},{"text":946,"config":947},"Privacy",{"href":768,"dataGaName":769,"dataGaLocation":711},[949],{"id":950,"title":7,"body":247,"config":951,"content":953,"description":247,"extension":957,"meta":958,"navigation":251,"path":959,"seo":960,"stem":961,"__hash__":962},"blogAuthors/en-us/blog/authors/dylan-griffith.yml",{"template":952},"BlogAuthor",{"name":7,"config":954},{"headshot":955,"ctfId":956},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749672822/Blog/Author%20Headshots/DylanGriffith-headshot.jpg","DylanGriffith","yml",{},"/en-us/blog/authors/dylan-griffith",{},"en-us/blog/authors/dylan-griffith","9wpc7kzdXwKMr4FZ6xU0rNn7VaQMStIIAbW69G1ZymY",[964,973,981],{"title":965,"description":966,"heroImage":967,"category":243,"date":968,"authors":969,"slug":972,"externalUrl":247},"Confidential AI for GitLab Self-Hosted","Give developers AI coding agents in GitLab Duo without source code leaving a hardware-encrypted boundary — no GPUs needed.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1773866173/vte9qh8rriznvyclhkes.png","2026-08-06",[970,971],"Mathias Ewald","Martin Paloncy, Edgeless Systems","confidential-ai-for-gitlab-self-hosted",{"title":974,"description":975,"heroImage":976,"category":243,"date":977,"authors":978,"slug":980,"externalUrl":247},"Green DevOps: Why carbon measurement belongs in your CI/CD pipeline","CI/CD pipelines have a hidden carbon cost. Here's why measuring it matters, and how you can get started with Eco CI and Carmen in GitLab.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1765809212/noh0mdfn9o94ry9ykura.png","2026-07-09",[979],"Lysanne Pinto","green-devops-carbon-measurement-cicd-pipeline",{"title":982,"description":983,"heroImage":984,"category":243,"date":985,"authors":986,"slug":988,"externalUrl":247},"How to build CI/CD observability at scale","This practical guide to GitLab pipeline analytics helps self-managed users gain operational insights using Prometheus and Grafana.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1774465167/n5hlvrsrheadeccyr1oz.png","2026-04-28",[987],"Paul Meresanu","how-to-build-ci-cd-observability-at-scale",{"promotions":990},[991,1005,1017,1029],{"id":992,"categories":993,"header":995,"text":996,"button":997,"image":1002},"ai-modernization",[994],"ai","Is AI achieving its promise at scale?","Quiz will take 5 minutes or less",{"text":998,"config":999},"Get your AI maturity score",{"href":1000,"dataGaName":1001,"dataGaLocation":476},"/assessments/ai-modernization-assessment/","modernization assessment",{"config":1003},{"src":1004},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138786/qix0m7kwnd8x2fh1zq49.png",{"id":1006,"categories":1007,"header":1009,"text":996,"button":1010,"image":1014},"devops-modernization",[1008,813],"product","Are you just managing tools or shipping innovation?",{"text":1011,"config":1012},"Get your DevOps maturity score",{"href":1013,"dataGaName":1001,"dataGaLocation":476},"/assessments/devops-modernization-assessment/",{"config":1015},{"src":1016},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138785/eg818fmakweyuznttgid.png",{"id":1018,"categories":1019,"header":1021,"text":996,"button":1022,"image":1026},"security-modernization",[1020],"security","Are you trading speed for security?",{"text":1023,"config":1024},"Get your security maturity score",{"href":1025,"dataGaName":1001,"dataGaLocation":476},"/assessments/security-modernization-assessment/",{"config":1027},{"src":1028},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138786/p4pbqd9nnjejg5ds6mdk.png",{"id":1030,"paths":1031,"header":1034,"text":1035,"button":1036,"image":1041},"github-azure-migration",[1032,1033],"migration-from-azure-devops-to-gitlab","integrating-azure-devops-scm-and-gitlab","Is your team ready for GitHub's Azure move?","GitHub is already rebuilding around Azure. Find out what it means for you.",{"text":1037,"config":1038},"See how GitLab compares to GitHub",{"href":1039,"dataGaName":1040,"dataGaLocation":476},"/compare/gitlab-vs-github/github-azure-migration/","github azure migration",{"config":1042},{"src":1016},{"header":1044,"blurb":1045,"button":1046,"secondaryButton":1051},"Start building faster today","See what your team can do with the intelligent orchestration platform for DevSecOps.\n",{"text":1047,"config":1048},"Get your free trial",{"href":1049,"dataGaName":275,"dataGaLocation":1050},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":749,"config":1052},{"href":601,"dataGaName":280,"dataGaLocation":1050},1786803736793]