[{"data":1,"prerenderedAt":1116},["ShallowReactive",2],{"/blog/path-to-decomposing-gitlab-database-part3":3,"navigation-en-us":328,"banner-en-us":756,"footer-en-us":766,"blog-post-authors-en-us-Dylan Griffith":1011,"blog-related-posts-en-us-path-to-decomposing-gitlab-database-part3":1026,"blog-promotions-en-us":1052,"next-steps-en-us":1106},{"id":4,"title":5,"authors":6,"body":8,"category":308,"date":309,"description":310,"extension":311,"externalUrl":312,"faq":312,"featured":313,"heroImage":314,"meta":315,"navigation":172,"path":316,"seo":317,"slug":321,"stem":322,"tags":323,"template":326,"updatedDate":312,"__hash__":327},"blogPosts/en-us/blog/path-to-decomposing-gitlab-database-part3.md","Decomposing the GitLab backend database, Part 3: Challenges and surprises",[7],"Dylan Griffith",{"type":9,"value":10,"toc":300},"minimark",[11,18,38,43,46,55,59,62,95,99,105,108,111,114,117,131,135,143,154,157,211,218,241,244,247,250,254,257,260,263,266,270,273,282,296],[12,13,14],"p",{},[15,16,17],"em",{},"This blog post is part 3 in a three-part series. It focuses on some interesting low-level challenges we faced along the way, as well as some surprises we found during the migration.",[19,20,21,31],"ul",{},[22,23,24,25,30],"li",{},"To read about the design and planning phase, check out ",[26,27,29],"a",{"href":28},"/blog/path-to-decomposing-gitlab-database-part1/","part\n1",".",[22,32,33,34,30],{},"To read about how we executed the actual migration and our results, check\nout ",[26,35,37],{"href":36},"/blog/path-to-decomposing-gitlab-database-part2/","part 2",[39,40,42],"h2",{"id":41},"the-challenge-with-taking-gitlabcom-offline","The challenge with taking GitLab.com offline",[12,44,45],{},"One key part of our migration process was to take all systems offline that could potentially talk to the database. This may seem as simple as \"shutting down the servers\" but given the scale and complexity of GitLab.com's infrastructure this proved to be really quite complex. Here is just a subset of the different things we had to shut down:",[47,48,49,52],"ol",{},[22,50,51],{},"Kubernetes pods corresponding to web, API, and Sidekiq services",[22,53,54],{},"Cron jobs across various VMs",[39,56,58],{"id":57},"surprises-along-the-way","Surprises along the way",[12,60,61],{},"Even though we had rehearsed the migration many times in staging, there were still some things that caught us off-guard in production. Luckily, we had allocated sufficient buffer time during the migration to resolve all of these during the call:",[47,63,64,86,89,92],{},[22,65,66,67,72,73,77,78,81,82,85],{},"Autovacuum on our largest CI tables take a long time and can run at any\ntime. This delayed our migration as we needed to gain table locks for our ",[26,68,71],{"href":69,"rel":70},"https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83211",[],"write block triggers",".\nAdding these triggers requires a ",[74,75,76],"code",{},"ShareRowExclusiveLock"," which cannot be acquired while the autovacuum is running for that table. We disabled some manual vacuum processes we were aware of ahead of the call but autovacuum can happen at any time and our ",[74,79,80],{},"ci_builds"," table just happen to have autovacuum at the time we were trying to block writes to this table. To work around this we needed to temporarily disable autovacuum for the relevant tables and then find the ",[74,83,84],{},"pid"," for the autovacuum process and terminate this which allowed our triggers to be successfully added.",[22,87,88],{},"Sometimes a long-running SSH session by an SRE or developer can leave\nopen a surprising database connection that needs to be tracked down and closed.",[22,90,91],{},"Cron jobs can be run on various hosts that start rails processes or\ndatabase connections at any time. We had many examples that were created with different purposes for database maintenance over the years, and we missed at least one in our practice runs. They weren't as easy to detect on staging as they may not all be configured on staging, or they run a lot faster on staging. Also, our staging runs all happened on week days, but our production migration happened on a weekend where it seemed we were deliberately running some database maintenance workloads during low utilization hours.",[22,93,94],{},"Our Sentry client-side error tracking caused us to overload our Sentry\nserver due to many of users leaving open GitLab browser tabs. As the browser tabs periodically make asynchronous requests to GitLab and get errors (since GitLab.com was down), they then send all these errors to Sentry and this overloaded our Sentry error server to the point we couldn't load it to check for errors. This was quickly diagnosed based on the URL all the requests were sent to, but it did delay our migration as checking for new errors was key to determining success or failure of the migration.",[39,96,98],{"id":97},"cascading-replication-doubles-latency-triples-in-our-case","Cascading replication doubles latency (triples in our case)",[12,100,101,102,30],{},"A key initial step in our phased rollout was to move all read-only CI traffic to dedicated CI replicas. These were cascading replicas from the main Patroni cluster. Furthermore, we made the decision to create the standby cluster leader as a replica of another replica in the Main Patroni cluster. Ultimately this meant the replication process for our CI replicas was ",[74,103,104],{},"Main Primary -> Main Replica -> CI Standby Leader -> CI Replica",[12,106,107],{},"This change meant that our CI replicas had roughly three times as much latency compared with our Main replicas, which previously served CI read-only traffic.",[12,109,110],{},"Since our read-only load balancing logic is based on users sticking to the primary until a replica catches up with the last write that they performed, users might end up sticking to the primary longer than they previously would have.",[12,112,113],{},"This may have served to increase our load on the primary database after rolling out Phase 3.",[12,115,116],{},"We never measured this impact, but in hindsight it is something we should have factored in and benchmarked with our gradual rollout of Phase 3.",[12,118,119,120,123,124,127,128,130],{},"Additionally, we should have considered mitigating this issue by having the ",[74,121,122],{},"CI Standby Leader"," replicating straight from the ",[74,125,126],{},"Main Primary"," or adding the ",[74,129,122],{}," to the pool of replicas that we could service CI read-only traffic.",[39,132,134],{"id":133},"re-balancing-pgbouncer-connections-incrementally-without-saturating","Re-balancing PGBouncer connections incrementally without saturating",[12,136,137,138,142],{},"anything [Phase 4 of our rollout]",[139,140,141],"span",{},"phase4_change_request"," turned out to be one of the trickiest parts of the migration. Since we wanted all phases (where possible)",[12,144,145,146,149,150,153],{},"to be rolled out incrementally we needed some way to [solve for incrementally re-balancing connection pool limits]",[139,147,148],{},"phase4_gradual_rollout_issue"," from ",[74,151,152],{},"GitLab -> PGBouncer -> Postgres"," without exceeding the total connection limit of Postgres or opening too many connections to Postgres that might saturate CPU. This was difficult because all the connection limits were very well tuned, and we were close to saturation across all these limits.",[12,155,156],{},"The gradual rollout of traffic for Phase 4 looked like:",[158,159,164],"pre",{"className":160,"code":161,"language":162,"meta":163,"style":163},"language-mermaid shiki shiki-themes github-light","\ngraph LR;\n    PostgresMain[(PostgresMain - Limit K max_connections)]\n    GitLabRails-->|100-X % of CI queries|PGBouncerMain\n    GitLabRails-->|X% of CI queries|PGBouncerCi\n    PGBouncerMain-->|Limit N pool_size|PostgresMain\n    PGBouncerCi-->|Limit M pool_size|PostgresMain\n","mermaid","",[74,165,166,174,181,187,193,199,205],{"__ignoreMap":163},[139,167,170],{"class":168,"line":169},"line",1,[139,171,173],{"emptyLinePlaceholder":172},true,"\n",[139,175,177],{"class":168,"line":176},2,[139,178,180],{"class":179},"sgsFI","graph LR;\n",[139,182,184],{"class":168,"line":183},3,[139,185,186],{"class":179},"    PostgresMain[(PostgresMain - Limit K max_connections)]\n",[139,188,190],{"class":168,"line":189},4,[139,191,192],{"class":179},"    GitLabRails-->|100-X % of CI queries|PGBouncerMain\n",[139,194,196],{"class":168,"line":195},5,[139,197,198],{"class":179},"    GitLabRails-->|X% of CI queries|PGBouncerCi\n",[139,200,202],{"class":168,"line":201},6,[139,203,204],{"class":179},"    PGBouncerMain-->|Limit N pool_size|PostgresMain\n",[139,206,208],{"class":168,"line":207},7,[139,209,210],{"class":179},"    PGBouncerCi-->|Limit M pool_size|PostgresMain\n",[12,212,213,214,217],{},"We wanted to gradually increase X from 0-100. But this presented a problem, because the number of connections to the ",[74,215,216],{},"PostgresMain"," DB will change with this number.",[12,219,220,221,224,225,228,229,232,233,236,237,240],{},"We assume it has some initial limit ",[74,222,223],{},"K"," connections, and we assume this limit is deliberately just high enough to handle the current connections from ",[74,226,227],{},"PGBouncerMain"," and not overload the CPU. We need to carefully tune ",[74,230,231],{},"N"," and ",[74,234,235],{},"M"," ",[74,238,239],{},"pool_size"," values across the separate PGBouncer processes to avoid overloading the limit K, and we also need to avoid saturating the",[12,242,243],{},"Postgres server CPU with too much traffic. At the same time, we need to ensure there are enough connections to handle the traffic to both PGBouncer pools.",[12,245,246],{},"We addressed this issue by taking very small steps during low utilization hours (where CPU and connection pools weren't near saturation)\nand doing very detailed analysis after each step. We would wait a day or so to figure out how many connections to move over with the following steps, based on the number of connections that were used by the smaller step. We also used what data we had early on from table-based metrics to get an insight into how many connections we thought we'd need to move to the CI PGBouncer pool.",[12,248,249],{},"In the end, we did need to make small adjustments to our estimates along the way as we saw saturation occur, but there was never any major user-facing saturation incidents, as the steps were small enough.",[39,251,253],{"id":252},"final-thoughts","Final thoughts",[12,255,256],{},"We're very happy with the results of this project overall.",[12,258,259],{},"A key objective of this project, which was hard to predict, was how the complexity of an additional database might impact developer productivity. They can't do certain types of joins and there is more information to be aware of.",[12,261,262],{},"However, many months have now passed, and it seems clear now that the complexity is mostly abstracted by Rails models. With continued large number of developers contributing, we have seen little-to-no impact on productivity.",[12,264,265],{},"Combining this success with the huge scalability headroom we've gained, we believe this was a great decision for GitLab.",[39,267,269],{"id":268},"more-reading","More reading",[12,271,272],{},"This blog series contains many links to see our early designing, planning, and implementation of various parts of this project.",[12,274,275,276,281],{},"GitLab's ",[26,277,280],{"href":278,"rel":279},"https://handbook.gitlab.com/handbook/values/#transparency",[],"transparency value"," means you can read all the details and get a sense of what it's like to work on projects like this at GitLab. If you'd like to know more or something was unclear, please leave a comment so we can make sure we share all our learnings.",[12,283,284,285,290,291,30],{},"You can read more details in ",[26,286,289],{"href":287,"rel":288},"https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6440",[],"the Phase 4 change request"," and in ",[26,292,295],{"href":293,"rel":294},"https://gitlab.com/gitlab-org/gitlab/-/issues/347203",[],"the Phase 4 gradual rollout issue",[297,298,299],"style",{},"html pre.shiki code .sgsFI, html code.shiki .sgsFI{--shiki-default:#24292E}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":163,"searchDepth":176,"depth":176,"links":301},[302,303,304,305,306,307],{"id":41,"depth":176,"text":42},{"id":57,"depth":176,"text":58},{"id":97,"depth":176,"text":98},{"id":133,"depth":176,"text":134},{"id":252,"depth":176,"text":253},{"id":268,"depth":176,"text":269},"engineering","2022-08-04","This is the final installment in our three-part series about our yearlong\nproject to decompose GitLab's Postgres database.","md",null,false,"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749663397/Blog/Hero%20Images/logoforblogpost.jpg",{},"/en-us/blog/path-to-decomposing-gitlab-database-part3",{"ogTitle":5,"ogImage":314,"ogDescription":310,"ogSiteName":318,"noIndex":313,"ogType":319,"ogUrl":320,"title":5,"canonicalUrls":320,"description":310},"https://about.gitlab.com","article","https://about.gitlab.com/blog/path-to-decomposing-gitlab-database-part3","path-to-decomposing-gitlab-database-part3","en-us/blog/path-to-decomposing-gitlab-database-part3",[324,325],"inside GitLab","design","BlogPost","N_PKKm4zyYQxSu4FhC_LJI-hAoX9mZlsxbSSXfxWZC4",{"logo":329,"freeTrial":334,"sales":339,"login":344,"items":349,"search":676,"minimal":707,"duo":726,"switchNav":735,"pricingDeployment":746},{"config":330},{"href":331,"dataGaName":332,"dataGaLocation":333},"/","gitlab logo","header",{"text":335,"config":336},"Get free trial",{"href":337,"dataGaName":338,"dataGaLocation":333},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":340,"config":341},"Request a demo",{"href":342,"dataGaName":343,"dataGaLocation":333},"/sales/?contact-topic=request-demo","sales",{"text":345,"config":346},"Sign in",{"href":347,"dataGaName":348,"dataGaLocation":333},"https://gitlab.com/users/sign_in/","sign in",[350,379,479,484,598,654],{"text":351,"config":352,"menu":354},"Platform",{"dataNavLevelOne":353},"platform",{"type":355,"columns":356},"cards",[357,363,371],{"title":351,"description":358,"link":359},"The intelligent orchestration platform for DevSecOps",{"text":360,"config":361},"Explore our Platform",{"href":362,"dataGaName":353,"dataGaLocation":333},"/platform/",{"title":364,"description":365,"link":366},"GitLab Duo Agent Platform","Agentic AI for the entire software lifecycle",{"text":367,"config":368},"Meet GitLab Duo",{"href":369,"dataGaName":370,"dataGaLocation":333},"/gitlab-duo-agent-platform/","gitlab duo agent platform",{"title":372,"description":373,"link":374},"Why GitLab","See the top reasons enterprises choose GitLab",{"text":375,"config":376},"Learn more",{"href":377,"dataGaName":378,"dataGaLocation":333},"/why-gitlab/","why gitlab",{"text":380,"left":172,"config":381,"menu":383},"Product",{"dataNavLevelOne":382},"solutions",{"type":384,"link":385,"columns":389,"feature":458},"lists",{"text":386,"config":387},"View all Solutions",{"href":388,"dataGaName":382,"dataGaLocation":333},"/solutions/",[390,414,437],{"title":391,"description":392,"link":393,"items":398},"Automation","CI/CD and automation to accelerate deployment",{"config":394},{"icon":395,"href":396,"dataGaName":397,"dataGaLocation":333},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[399,403,406,410],{"text":400,"config":401},"CI/CD",{"href":402,"dataGaLocation":333,"dataGaName":400},"/solutions/continuous-integration/",{"text":364,"config":404},{"href":369,"dataGaLocation":333,"dataGaName":405},"gitlab duo agent platform - product menu",{"text":407,"config":408},"Source Code Management",{"href":409,"dataGaLocation":333,"dataGaName":407},"/solutions/source-code-management/",{"text":411,"config":412},"Automated Software Delivery",{"href":396,"dataGaLocation":333,"dataGaName":413},"Automated software delivery",{"title":415,"description":416,"link":417,"items":422},"Security","Deliver code faster without compromising security",{"config":418},{"href":419,"dataGaName":420,"dataGaLocation":333,"icon":421},"/solutions/application-security-testing/","security and compliance","ShieldCheckLight",[423,427,432],{"text":424,"config":425},"Application Security Testing",{"href":419,"dataGaName":426,"dataGaLocation":333},"Application security testing",{"text":428,"config":429},"Software Supply Chain Security",{"href":430,"dataGaLocation":333,"dataGaName":431},"/solutions/supply-chain/","Software supply chain security",{"text":433,"config":434},"Software Compliance",{"href":435,"dataGaName":436,"dataGaLocation":333},"/solutions/software-compliance/","software compliance",{"title":438,"link":439,"items":444},"Measurement",{"config":440},{"icon":441,"href":442,"dataGaName":443,"dataGaLocation":333},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[445,449,453],{"text":446,"config":447},"Visibility & Measurement",{"href":442,"dataGaLocation":333,"dataGaName":448},"Visibility and Measurement",{"text":450,"config":451},"Value Stream Management",{"href":452,"dataGaLocation":333,"dataGaName":450},"/solutions/value-stream-management/",{"text":454,"config":455},"Analytics & Insights",{"href":456,"dataGaLocation":333,"dataGaName":457},"/solutions/analytics-and-insights/","Analytics and insights",{"title":459,"type":384,"items":460},"GitLab for",[461,467,473],{"text":462,"config":463},"Enterprise",{"icon":464,"href":465,"dataGaLocation":333,"dataGaName":466},"Building","/enterprise/","enterprise",{"text":468,"config":469},"Small Business",{"icon":470,"href":471,"dataGaLocation":333,"dataGaName":472},"Work","/small-business/","small business",{"text":474,"config":475},"Public Sector",{"icon":476,"href":477,"dataGaLocation":333,"dataGaName":478},"Organization","/solutions/public-sector/","public sector",{"text":480,"config":481},"Pricing",{"href":482,"dataGaName":483,"dataGaLocation":333,"dataNavLevelOne":483},"/pricing/","pricing",{"text":485,"config":486,"menu":488},"Resources",{"dataNavLevelOne":487},"resources",{"type":384,"link":489,"columns":493,"feature":587},{"text":490,"config":491},"View all resources",{"href":492,"dataGaName":487,"dataGaLocation":333},"/resources/",[494,527,554],{"title":495,"items":496},"Getting started",[497,502,507,512,517,522],{"text":498,"config":499},"Install",{"href":500,"dataGaName":501,"dataGaLocation":333},"/install/","install",{"text":503,"config":504},"Quick start guides",{"href":505,"dataGaName":506,"dataGaLocation":333},"/get-started/","quick setup checklists",{"text":508,"config":509},"Learn",{"href":510,"dataGaLocation":333,"dataGaName":511},"https://university.gitlab.com/","learn",{"text":513,"config":514},"Product documentation",{"href":515,"dataGaName":516,"dataGaLocation":333},"https://docs.gitlab.com/","product documentation",{"text":518,"config":519},"Best practice videos",{"href":520,"dataGaName":521,"dataGaLocation":333},"/getting-started-videos/","best practice videos",{"text":523,"config":524},"Integrations",{"href":525,"dataGaName":526,"dataGaLocation":333},"/integrations/","integrations",{"title":528,"items":529},"Discover",[530,535,540,545,549],{"text":531,"config":532},"Customer success stories",{"href":533,"dataGaName":534,"dataGaLocation":333},"/customers/","customer success stories",{"text":536,"config":537},"Blog",{"href":538,"dataGaName":539,"dataGaLocation":333},"/blog/","blog",{"text":541,"config":542},"Demo Hub",{"href":543,"dataGaName":544,"dataGaLocation":333},"/demo-hub/","demo hub",{"text":546,"config":547},"The Source",{"href":548,"dataGaName":539,"dataGaLocation":333},"/the-source/",{"text":550,"config":551},"Remote",{"href":552,"dataGaName":553,"dataGaLocation":333},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"title":555,"items":556},"Connect",[557,562,567,572,577,582],{"text":558,"config":559},"GitLab Services",{"href":560,"dataGaName":561,"dataGaLocation":333},"/services/","services",{"text":563,"config":564},"Contribute",{"href":565,"dataGaName":566,"dataGaLocation":333},"https://contributors.gitlab.com","contribute",{"text":568,"config":569},"Community",{"href":570,"dataGaName":571,"dataGaLocation":333},"/community/","community",{"text":573,"config":574},"Forum",{"href":575,"dataGaName":576,"dataGaLocation":333},"https://forum.gitlab.com/","forum",{"text":578,"config":579},"Events",{"href":580,"dataGaName":581,"dataGaLocation":333},"/events/","events",{"text":583,"config":584},"Partners",{"href":585,"dataGaName":586,"dataGaLocation":333},"/partners/","partners",{"config":588,"title":591,"text":592,"link":593},{"background":589,"textColor":590},"url('https://res.cloudinary.com/about-gitlab-com/image/upload/v1777322348/qpq8yrgn8knii57omj0c.png')","#000","What’s new in GitLab","Stay updated with our latest features and improvements.",{"text":594,"config":595},"Read the latest",{"href":596,"dataGaName":597,"dataGaLocation":333},"/whats-new/","whats new",{"text":599,"config":600,"menu":602},"Company",{"dataNavLevelOne":601},"company",{"type":384,"columns":603},[604],{"items":605},[606,611,617,619,624,629,634,639,644,649],{"text":607,"config":608},"About",{"href":609,"dataGaName":610,"dataGaLocation":333},"/company/","about",{"text":612,"config":613,"footerGa":616},"Jobs",{"href":614,"dataGaName":615,"dataGaLocation":333},"/jobs/","jobs",{"dataGaName":615},{"text":578,"config":618},{"href":580,"dataGaName":581,"dataGaLocation":333},{"text":620,"config":621},"Leadership",{"href":622,"dataGaName":623,"dataGaLocation":333},"/company/team/e-group/","leadership",{"text":625,"config":626},"Handbook",{"href":627,"dataGaName":628,"dataGaLocation":333},"https://handbook.gitlab.com/","handbook",{"text":630,"config":631},"Investor relations",{"href":632,"dataGaName":633,"dataGaLocation":333},"https://ir.gitlab.com/overview/default.aspx","investor relations",{"text":635,"config":636},"Trust Center",{"href":637,"dataGaName":638,"dataGaLocation":333},"/security/","trust center",{"text":640,"config":641},"AI Transparency Center",{"href":642,"dataGaName":643,"dataGaLocation":333},"/ai-transparency-center/","ai transparency center",{"text":645,"config":646},"Newsletter",{"href":647,"dataGaName":648,"dataGaLocation":333},"/company/contact/#contact-forms","newsletter",{"text":650,"config":651},"Press",{"href":652,"dataGaName":653,"dataGaLocation":333},"/press/","press",{"text":655,"config":656,"menu":657},"Contact us",{"dataNavLevelOne":601},{"type":384,"columns":658},[659],{"items":660},[661,666,671],{"text":662,"config":663},"Talk to sales",{"href":664,"dataGaName":665,"dataGaLocation":333},"/sales/","talk to sales",{"text":667,"config":668},"Support portal",{"href":669,"dataGaName":670,"dataGaLocation":333},"https://support.gitlab.com/hc/en-us","support portal",{"text":672,"config":673},"Customer portal",{"href":674,"dataGaName":675,"dataGaLocation":333},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":677,"login":678,"suggestions":685},"Close",{"text":679,"link":680},"To search repositories and projects, login to",{"text":681,"config":682},"gitlab.com",{"href":347,"dataGaName":683,"dataGaLocation":684},"search login","search",{"text":686,"default":687},"Suggestions",[688,690,694,696,700,704],{"text":364,"config":689},{"href":369,"dataGaName":364,"dataGaLocation":684},{"text":691,"config":692},"Code Suggestions (AI)",{"href":693,"dataGaName":691,"dataGaLocation":684},"/solutions/code-suggestions/",{"text":400,"config":695},{"href":402,"dataGaName":400,"dataGaLocation":684},{"text":697,"config":698},"GitLab on AWS",{"href":699,"dataGaName":697,"dataGaLocation":684},"/partners/technology-partners/aws/",{"text":701,"config":702},"GitLab on Google Cloud",{"href":703,"dataGaName":701,"dataGaLocation":684},"/partners/technology-partners/google-cloud-platform/",{"text":705,"config":706},"Why GitLab?",{"href":377,"dataGaName":705,"dataGaLocation":684},{"freeTrial":708,"mobileIcon":713,"desktopIcon":718,"secondaryButton":721},{"text":709,"config":710},"Start free trial",{"href":711,"dataGaName":338,"dataGaLocation":712},"https://gitlab.com/-/trials/new/","nav",{"altText":714,"config":715},"Gitlab Icon",{"src":716,"dataGaName":717,"dataGaLocation":712},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":714,"config":719},{"src":720,"dataGaName":717,"dataGaLocation":712},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":722,"config":723},"Get Started",{"href":724,"dataGaName":725,"dataGaLocation":712},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/get-started/","get started",{"freeTrial":727,"mobileIcon":731,"desktopIcon":733},{"text":728,"config":729},"Learn more about GitLab Duo",{"href":369,"dataGaName":730,"dataGaLocation":712},"gitlab duo",{"altText":714,"config":732},{"src":716,"dataGaName":717,"dataGaLocation":712},{"altText":714,"config":734},{"src":720,"dataGaName":717,"dataGaLocation":712},{"button":736,"mobileIcon":741,"desktopIcon":743},{"text":737,"config":738},"/switch",{"href":739,"dataGaName":740,"dataGaLocation":712},"#contact","switch",{"altText":714,"config":742},{"src":716,"dataGaName":717,"dataGaLocation":712},{"altText":714,"config":744},{"src":745,"dataGaName":717,"dataGaLocation":712},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1773335277/ohhpiuoxoldryzrnhfrh.png",{"freeTrial":747,"mobileIcon":752,"desktopIcon":754},{"text":748,"config":749},"Back to pricing",{"href":482,"dataGaName":750,"dataGaLocation":712,"icon":751},"back to pricing","GoBack",{"altText":714,"config":753},{"src":716,"dataGaName":717,"dataGaLocation":712},{"altText":714,"config":755},{"src":720,"dataGaName":717,"dataGaLocation":712},{"title":757,"titleMobile":758,"button":759,"config":764},"Duo Agent Platform delivers 400% ROI, per new Forrester Consulting study.","400% ROI: Forrester TEI for GitLab Duo",{"text":375,"config":760},{"href":761,"dataGaName":762,"dataGaLocation":763},"https://about.gitlab.com/blog/gitlab-duo-agent-platform-delivers-400-percent-roi/","forrester-tei-dap-banner","global-banner",{"layout":765,"disabled":313},"release",{"data":767},{"text":768,"source":769,"edit":775,"contribute":780,"config":785,"items":790,"minimal":1000},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":770,"config":771},"View page source",{"href":772,"dataGaName":773,"dataGaLocation":774},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":776,"config":777},"Edit this page",{"href":778,"dataGaName":779,"dataGaLocation":774},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":781,"config":782},"Please contribute",{"href":783,"dataGaName":784,"dataGaLocation":774},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":786,"facebook":787,"youtube":788,"linkedin":789},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[791,838,892,936,968],{"title":480,"links":792,"subMenu":807},[793,797,802],{"text":794,"config":795},"View plans",{"href":482,"dataGaName":796,"dataGaLocation":774},"view plans",{"text":798,"config":799},"Why Premium?",{"href":800,"dataGaName":801,"dataGaLocation":774},"/pricing/premium/","why premium",{"text":803,"config":804},"Why Ultimate?",{"href":805,"dataGaName":806,"dataGaLocation":774},"/pricing/ultimate/","why ultimate",[808],{"title":809,"links":810},"Contact Us",[811,814,816,818,823,828,833],{"text":812,"config":813},"Contact sales",{"href":664,"dataGaName":343,"dataGaLocation":774},{"text":667,"config":815},{"href":669,"dataGaName":670,"dataGaLocation":774},{"text":672,"config":817},{"href":674,"dataGaName":675,"dataGaLocation":774},{"text":819,"config":820},"Status",{"href":821,"dataGaName":822,"dataGaLocation":774},"https://status.gitlab.com/","status",{"text":824,"config":825},"Terms of use",{"href":826,"dataGaName":827,"dataGaLocation":774},"/terms/","terms of use",{"text":829,"config":830},"Privacy statement",{"href":831,"dataGaName":832,"dataGaLocation":774},"/privacy/","privacy statement",{"text":834,"config":835},"Cookie preferences",{"dataGaName":836,"dataGaLocation":774,"id":837,"isOneTrustButton":172},"cookie preferences","ot-sdk-btn",{"title":380,"links":839,"subMenu":848},[840,844],{"text":841,"config":842},"DevSecOps platform",{"href":362,"dataGaName":843,"dataGaLocation":774},"devsecops platform",{"text":845,"config":846},"AI-Assisted Development",{"href":369,"dataGaName":847,"dataGaLocation":774},"ai-assisted development",[849],{"title":850,"links":851},"Topics",[852,857,862,867,872,877,882,887],{"text":853,"config":854},"CICD",{"href":855,"dataGaName":856,"dataGaLocation":774},"/topics/ci-cd/","cicd",{"text":858,"config":859},"GitOps",{"href":860,"dataGaName":861,"dataGaLocation":774},"/topics/gitops/","gitops",{"text":863,"config":864},"DevOps",{"href":865,"dataGaName":866,"dataGaLocation":774},"/topics/devops/","devops",{"text":868,"config":869},"Version Control",{"href":870,"dataGaName":871,"dataGaLocation":774},"/topics/version-control/","version control",{"text":873,"config":874},"DevSecOps",{"href":875,"dataGaName":876,"dataGaLocation":774},"/topics/devsecops/","devsecops",{"text":878,"config":879},"Cloud Native",{"href":880,"dataGaName":881,"dataGaLocation":774},"/topics/cloud-native/","cloud native",{"text":883,"config":884},"AI for Coding",{"href":885,"dataGaName":886,"dataGaLocation":774},"/topics/devops/ai-for-coding/","ai for coding",{"text":888,"config":889},"Agentic AI",{"href":890,"dataGaName":891,"dataGaLocation":774},"/topics/agentic-ai/","agentic ai",{"title":893,"links":894},"Solutions",[895,897,899,904,908,911,915,918,920,923,926,931],{"text":424,"config":896},{"href":419,"dataGaName":424,"dataGaLocation":774},{"text":413,"config":898},{"href":396,"dataGaName":397,"dataGaLocation":774},{"text":900,"config":901},"Agile development",{"href":902,"dataGaName":903,"dataGaLocation":774},"/solutions/agile-delivery/","agile delivery",{"text":905,"config":906},"SCM",{"href":409,"dataGaName":907,"dataGaLocation":774},"source code management",{"text":853,"config":909},{"href":402,"dataGaName":910,"dataGaLocation":774},"continuous integration & delivery",{"text":912,"config":913},"Value stream management",{"href":452,"dataGaName":914,"dataGaLocation":774},"value stream management",{"text":858,"config":916},{"href":917,"dataGaName":861,"dataGaLocation":774},"/solutions/gitops/",{"text":462,"config":919},{"href":465,"dataGaName":466,"dataGaLocation":774},{"text":921,"config":922},"Small business",{"href":471,"dataGaName":472,"dataGaLocation":774},{"text":924,"config":925},"Public sector",{"href":477,"dataGaName":478,"dataGaLocation":774},{"text":927,"config":928},"Education",{"href":929,"dataGaName":930,"dataGaLocation":774},"/solutions/education/","education",{"text":932,"config":933},"Financial services",{"href":934,"dataGaName":935,"dataGaLocation":774},"/solutions/finance/","financial services",{"title":485,"links":937},[938,940,942,944,947,949,952,954,956,958,960,962,964,966],{"text":498,"config":939},{"href":500,"dataGaName":501,"dataGaLocation":774},{"text":503,"config":941},{"href":505,"dataGaName":506,"dataGaLocation":774},{"text":508,"config":943},{"href":510,"dataGaName":511,"dataGaLocation":774},{"text":513,"config":945},{"href":515,"dataGaName":946,"dataGaLocation":774},"docs",{"text":536,"config":948},{"href":538,"dataGaName":539,"dataGaLocation":774},{"text":950,"config":951},"What's new",{"href":596,"dataGaName":597,"dataGaLocation":774},{"text":531,"config":953},{"href":533,"dataGaName":534,"dataGaLocation":774},{"text":550,"config":955},{"href":552,"dataGaName":553,"dataGaLocation":774},{"text":558,"config":957},{"href":560,"dataGaName":561,"dataGaLocation":774},{"text":563,"config":959},{"href":565,"dataGaName":566,"dataGaLocation":774},{"text":568,"config":961},{"href":570,"dataGaName":571,"dataGaLocation":774},{"text":573,"config":963},{"href":575,"dataGaName":576,"dataGaLocation":774},{"text":578,"config":965},{"href":580,"dataGaName":581,"dataGaLocation":774},{"text":583,"config":967},{"href":585,"dataGaName":586,"dataGaLocation":774},{"title":599,"links":969},[970,972,974,976,978,980,984,989,991,993,995],{"text":607,"config":971},{"href":609,"dataGaName":601,"dataGaLocation":774},{"text":612,"config":973},{"href":614,"dataGaName":615,"dataGaLocation":774},{"text":620,"config":975},{"href":622,"dataGaName":623,"dataGaLocation":774},{"text":625,"config":977},{"href":627,"dataGaName":628,"dataGaLocation":774},{"text":630,"config":979},{"href":632,"dataGaName":633,"dataGaLocation":774},{"text":981,"config":982},"Sustainability",{"href":983,"dataGaName":981,"dataGaLocation":774},"/sustainability/",{"text":985,"config":986},"Diversity, inclusion and belonging (DIB)",{"href":987,"dataGaName":988,"dataGaLocation":774},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":635,"config":990},{"href":637,"dataGaName":638,"dataGaLocation":774},{"text":645,"config":992},{"href":647,"dataGaName":648,"dataGaLocation":774},{"text":650,"config":994},{"href":652,"dataGaName":653,"dataGaLocation":774},{"text":996,"config":997},"Modern Slavery Transparency Statement",{"href":998,"dataGaName":999,"dataGaLocation":774},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"items":1001},[1002,1005,1008],{"text":1003,"config":1004},"Terms",{"href":826,"dataGaName":827,"dataGaLocation":774},{"text":1006,"config":1007},"Cookies",{"dataGaName":836,"dataGaLocation":774,"id":837,"isOneTrustButton":172},{"text":1009,"config":1010},"Privacy",{"href":831,"dataGaName":832,"dataGaLocation":774},[1012],{"id":1013,"title":7,"body":312,"config":1014,"content":1016,"description":312,"extension":1020,"meta":1021,"navigation":172,"path":1022,"seo":1023,"stem":1024,"__hash__":1025},"blogAuthors/en-us/blog/authors/dylan-griffith.yml",{"template":1015},"BlogAuthor",{"name":7,"config":1017},{"headshot":1018,"ctfId":1019},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749672822/Blog/Author%20Headshots/DylanGriffith-headshot.jpg","DylanGriffith","yml",{},"/en-us/blog/authors/dylan-griffith",{},"en-us/blog/authors/dylan-griffith","9wpc7kzdXwKMr4FZ6xU0rNn7VaQMStIIAbW69G1ZymY",[1027,1036,1044],{"title":1028,"description":1029,"heroImage":1030,"category":308,"date":1031,"authors":1032,"slug":1035,"externalUrl":312},"Confidential AI for GitLab Self-Hosted","Give developers AI coding agents in GitLab Duo without source code leaving a hardware-encrypted boundary — no GPUs needed.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1773866173/vte9qh8rriznvyclhkes.png","2026-08-06",[1033,1034],"Mathias Ewald","Martin Paloncy, Edgeless Systems","confidential-ai-for-gitlab-self-hosted",{"title":1037,"description":1038,"heroImage":1039,"category":308,"date":1040,"authors":1041,"slug":1043,"externalUrl":312},"Green DevOps: Why carbon measurement belongs in your CI/CD pipeline","CI/CD pipelines have a hidden carbon cost. Here's why measuring it matters, and how you can get started with Eco CI and Carmen in GitLab.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1765809212/noh0mdfn9o94ry9ykura.png","2026-07-09",[1042],"Lysanne Pinto","green-devops-carbon-measurement-cicd-pipeline",{"title":1045,"description":1046,"heroImage":1047,"category":308,"date":1048,"authors":1049,"slug":1051,"externalUrl":312},"How to build CI/CD observability at scale","This practical guide to GitLab pipeline analytics helps self-managed users gain operational insights using Prometheus and Grafana.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1774465167/n5hlvrsrheadeccyr1oz.png","2026-04-28",[1050],"Paul Meresanu","how-to-build-ci-cd-observability-at-scale",{"promotions":1053},[1054,1068,1080,1092],{"id":1055,"categories":1056,"header":1058,"text":1059,"button":1060,"image":1065},"ai-modernization",[1057],"ai","Is AI achieving its promise at scale?","Quiz will take 5 minutes or less",{"text":1061,"config":1062},"Get your AI maturity score",{"href":1063,"dataGaName":1064,"dataGaLocation":539},"/assessments/ai-modernization-assessment/","modernization assessment",{"config":1066},{"src":1067},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138786/qix0m7kwnd8x2fh1zq49.png",{"id":1069,"categories":1070,"header":1072,"text":1059,"button":1073,"image":1077},"devops-modernization",[1071,876],"product","Are you just managing tools or shipping innovation?",{"text":1074,"config":1075},"Get your DevOps maturity score",{"href":1076,"dataGaName":1064,"dataGaLocation":539},"/assessments/devops-modernization-assessment/",{"config":1078},{"src":1079},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138785/eg818fmakweyuznttgid.png",{"id":1081,"categories":1082,"header":1084,"text":1059,"button":1085,"image":1089},"security-modernization",[1083],"security","Are you trading speed for security?",{"text":1086,"config":1087},"Get your security maturity score",{"href":1088,"dataGaName":1064,"dataGaLocation":539},"/assessments/security-modernization-assessment/",{"config":1090},{"src":1091},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138786/p4pbqd9nnjejg5ds6mdk.png",{"id":1093,"paths":1094,"header":1097,"text":1098,"button":1099,"image":1104},"github-azure-migration",[1095,1096],"migration-from-azure-devops-to-gitlab","integrating-azure-devops-scm-and-gitlab","Is your team ready for GitHub's Azure move?","GitHub is already rebuilding around Azure. Find out what it means for you.",{"text":1100,"config":1101},"See how GitLab compares to GitHub",{"href":1102,"dataGaName":1103,"dataGaLocation":539},"/compare/gitlab-vs-github/github-azure-migration/","github azure migration",{"config":1105},{"src":1079},{"header":1107,"blurb":1108,"button":1109,"secondaryButton":1114},"Start building faster today","See what your team can do with the intelligent orchestration platform for DevSecOps.\n",{"text":1110,"config":1111},"Get your free trial",{"href":1112,"dataGaName":338,"dataGaLocation":1113},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":812,"config":1115},{"href":664,"dataGaName":343,"dataGaLocation":1113},1786803751840]