[{"data":1,"prerenderedAt":1241},["ShallowReactive",2],{"/blog/specialized-sidekiq-configuration-lessons-from-gitlab-dot-com":3,"navigation-en-us":453,"banner-en-us":881,"footer-en-us":891,"blog-post-authors-en-us-Craig Miskell":1136,"blog-related-posts-en-us-specialized-sidekiq-configuration-lessons-from-gitlab-dot-com":1151,"blog-promotions-en-us":1177,"next-steps-en-us":1231},{"id":4,"title":5,"authors":6,"body":8,"category":435,"date":436,"description":437,"extension":438,"externalUrl":439,"faq":439,"featured":440,"heroImage":441,"meta":442,"navigation":443,"path":444,"seo":445,"slug":449,"stem":450,"tags":439,"template":451,"updatedDate":439,"__hash__":452},"blogPosts/en-us/blog/specialized-sidekiq-configuration-lessons-from-gitlab-dot-com.md","What we learned about configuring Sidekiq from GitLab.com",[7],"Craig Miskell",{"type":9,"value":10,"toc":420},"minimark",[11,15,18,32,39,44,47,57,60,80,83,88,91,100,106,109,112,118,122,126,144,151,154,158,161,169,172,175,178,186,194,197,212,225,229,238,241,248,262,269,276,279,291,298,302,311,320,338,355,378,381,388,392,401,406],[12,13,14],"p",{},"Sidekiq in GitLab works perfectly well out-of-the-box in most use cases, but it requires a little more attention in larger deployments or other specialized cases. We've learned a lot about how to configure Sidekiq in large deployments by maintaining GitLab.com – one of the largest instances of GitLab in existence. We added some critical features to GitLab.com in the past year to make it easier to configure Sidekiq in a manner more consistent with the maintainer's guidance, having strayed from this path for some time.",[12,16,17],{},"We are publishing two blog posts devoted to this topic. In this first post, we will unpack how we configured Sidekiq for GitLab.com, and in our second post, we will explain how to apply this to your GitLab instance.",[12,19,20],{},[21,22,23],"strong",{},[24,25,26,27],"span",{},"Learn more about ",[28,29,31],"a",{"href":30},"/blog/scaling-our-use-of-sidekiq/","how we iterated on Sidekiq background jobs",[12,33,34,35,38],{},"We built on that work and the learnings for the project we describe in a ",[28,36,37],{"href":30},"blog post on Sidekiq background jobs",".",[40,41,43],"h2",{"id":42},"what-is-sidekiq","What is Sidekiq?",[12,45,46],{},"Sidekiq is usually the background job processor of choice for Ruby-on-Rails, and uses Redis as a data store for the job queues. Background (or asynchronous) job processing is critical to GitLab because there are many\ntasks that:",[48,49,50,54],"ol",{},[51,52,53],"li",{},"Shouldn't tie up relatively expensive HTTP workers to perform long-running operations",[51,55,56],{},"Do not operate within an HTTP-request context (e.g., scheduled/periodic tasks)",[12,58,59],{},"For most users, how Sidekiq uses Redis doesn't matter much – Sidekiq receives a Redis connection and magic ensues, but at larger scales it becomes important.",[12,61,62,63,68,69,74,75,38],{},"The ",[28,64,67],{"href":65,"rel":66},"https://redis.io/commands#list",[],"Redis data structure that Sidekiq uses for queues is a LIST"," which is literally an\nordered sequence of entries. For Sidekiq, each entry is some JSON which describes the work to do (Ruby class + arguments)\nand some metadata. Out-of-the-box Sidekiq uses a single queue named \"default,\" but it's possible to create and use any\nnumber of other named queues if there is at least one Sidekiq worker configured to look at every queue. Jobs are enqueued at the end of the list using ",[28,70,73],{"href":71,"rel":72},"https://redis.io/commands/rpush",[],"RPUSH",", and are retrieved for execution from the front of the list with ",[28,76,79],{"href":77,"rel":78},"https://redis.io/commands/brpop",[],"BRPOP",[12,81,82],{},"A key fact is that BRPOP is a blocking operation – a Sidekiq worker looking to perform work will issue and be blocked by a single BRPOP until any work is available or a timeout (2-second default) is exceeded. Redis then returns the job (if available) and removes it from the LIST.",[84,85,87],"h3",{"id":86},"about-concurrency-threading","About Concurrency (threading)",[12,89,90],{},"This is a little bit tangential, but is a important later, so bear with me (or skip this section and come back to it later if you really need it).",[12,92,93,94,99],{},"When starting Sidekiq you can tell it how many threads to run, and where each thread request works from Redis and can\npotentially be executing a job. Sounds like an easy way to allow Sidekiq to do more work, right? Well, not exactly, because threading in Ruby is subject to the Global Interpreter Lock (GIL).\nFor more, ",[28,95,98],{"href":96,"rel":97},"https://thoughtbot.com/blog/untangling-ruby-threads",[],"read this great explanation about threading",", from which I will quote one key statement:",[101,102,103],"blockquote",{},[12,104,105],{},"This means that no matter how many threads you spawn, and how many cores you have at your disposal, MRI will literally\nnever be executing Ruby code in multiple threads concurrently",[12,107,108],{},"So each Sidekiq worker process will – at best – only occupy one CPU. Threading is about avoiding constraints on\nthroughput from blocking I/O (mostly network, like Web requests or DB queries).",[12,110,111],{},"The default concurrency is 25, which is fine for a default GitLab installation on a single node with a single Sidekiq\nworker and a wide mix of jobs. But if the jobs are mostly CPU-bound (doing heavy CPU computation in Ruby) then 25 may\nbe far too high and counter-productive as threads compete for the GIL. Or, if your workload is heavily network\ndependent, a higher number might be acceptable since most of the time is spent waiting.",[12,113,114,117],{},[21,115,116],{},"Why does this matter?"," When you start splitting up (sharding) your Sidekiq fleet, you need to pay attention to what\nconcurrency you give to each shard to ensure it is compatible with the subset of Sidekiq jobs that will be executing here.",[40,119,121],{"id":120},"how-we-configured-sidekiq-on-gitlabcom","How we configured Sidekiq on GitLab.com",[84,123,125],{"id":124},"how-we-historically-used-sidekiq","How we historically used Sidekiq",[12,127,128,129,134,135,140,141,38],{},"Some time ago in ",[28,130,133],{"href":131,"rel":132},"https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7006",[],"GitLab history",", we\ndecided to use one Sidekiq queue per worker (class), with the name of the queue automatically derived from the class\n(name + metadata), e.g., the class WebHookWorker runs in the web_hook queue. This approach has some benefits, but the author of\nSidekiq ",[28,136,139],{"href":137,"rel":138},"https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues",[],"does not recommend"," ",[21,142,143],{},"having more than a \"handful\" of queues per Sidekiq process",[12,145,146,147,150],{},"I assume handful means around 10 queues. At the time, we had about 45 job classes/queues which was beyond a \"handful\"\nbut not excessively so. However, as the GitLab code base has grown, we've added more job classes and queues. Currently, we have 440, and more will inevitably be added as new features are added. As discussed in the ",[28,148,149],{"href":30},"previous blog post",", we split our Sidekiq worker fleet into multiple shards with different collections of jobs on each shard, based on the job resource requirements and impact on user experience.",[12,152,153],{},"However, our \"catchall\" shard is still responsible for roughly 300 of those workers/queues. So each time a catchall Sidekiq worker requests the next available job, it issues a BRPOP with a huge list of queues. Redis then needs to parse that request, set up internal data structures for each queue, find the next available job, and then tear down those data structures. That's a lot of overhead just to fetch one job. We knew this was going to be a problem eventually, but for a while we were able to put our effort into other areas.",[84,155,157],{"id":156},"how-we-configured-sidekiq-on-gitlabcom-today","How we configured Sidekiq on GitLab.com today",[12,159,160],{},"By early-to-mid 2021, the Redis instance dedicated to Sidekiq was starting to hit more than 95% CPU saturation at peak:",[12,162,163,168],{},[164,165],"img",{"alt":166,"src":167},"Redis CPU Saturation","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782399026/blog/Content%20Images/sidekiq-2021/redis-cpu-saturation.png","\nWhat it looks like when Redis CPU usage reaches 95%",[12,170,171],{},"Redis is fundamentally single-threaded. Sure, IO Threads in version 6 changes that a bit, but command execution is serialized\nthrough the core thread, so once utilization hits 100% of a CPU core, it doesn't matter how many other idle/spare CPUs you have.",[12,173,174],{},"In my opinion, Redis is an absolutely amazing bit of software – the documentation is excellent, it is more robust than we deserve, and the throughput is spectacular on a single core. It has carried us a long way, but there is this hard limit that we cannot pass.",[12,176,177],{},"If we do exceed these limits, Sidekiq work will not be dispatched fast enough at peak times, and things will go wrong – possibly in quite subtle and troublesome ways. Last year, we gained a lot of headroom by upgrading to the latest CPUs in GCP, but that's not repeatable and merely put off the inevitable. BRPOP with many queues is the core reason for this saturation with all that overhead on every request from thousands of Sidekiq workers. So what else could we do?",[12,179,180,181,185],{},"As we understand it, the CPU usage is generated by a combination of the number of queues ",[182,183,184],"em",{},"and"," the number of workers listening to those queues for work, so we had two possible paths ahead of us:",[48,187,188,191],{},[51,189,190],{},"Reduce the number of workers using a given Redis by splitting Sidekiq into multiple fleets",[51,192,193],{},"Keep a single logical fleet and reduce the number of queues",[12,195,196],{},"Both were plausible options for reducing Redis CPU usage, but didn't overlap in implementation, and had quite distinct challenges.",[12,198,199,200,205,206,211],{},"We needed more data in order to make the best choice. We ",[28,201,204],{"href":202,"rel":203},"https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/956",[],"performed some experiments"," by spending a few days creating an artificial test harness that produced and consumed Sidekiq jobs at volumes mimicking what we see on GitLab.com. I cannot emphasize enough how artificial the workload is, and although we added some complexity to replicate certain aspects of production, it will never be the same as the real workload on GitLab.com. It did ",[28,207,210],{"href":208,"rel":209},"https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/956#note_542558210",[],"show that reducing the number of queues to \"one queue per shard\" had the greatest effect",". Other discussions also concluded this approach was likely safer, so the decision was easy to make.",[12,213,214,215,220,221,224],{},"If you're interested, ",[28,216,219],{"href":217,"rel":218},"https://gitlab.com/gitlab-com/gl-infra/sidekiq-redis-experiments/",[],"the code for those experiments is available here",", but fair warning, it is ",[182,222,223],{},"just"," enough to do what we needed to do, and requires some manual setup.",[84,226,228],{"id":227},"how-we-adjusted-the-sidekiq-routing-rules","How we adjusted the Sidekiq routing rules",[12,230,231,232,237],{},"This change would move us away from the one-queue-per-worker paradigm, but we still need to maintain the current\nsharding configuration on GitLab.com in particular, so our Sidekiq configuration is a fine balance of queues and workers\nand we cannot throw that all away. So we picked up work from ",[28,233,236],{"href":234,"rel":235},"https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/194",[],"last year",", adjusted the plan slightly, and implemented Sidekiq routing rules.",[12,239,240],{},"Prior to routing rules, the decision on where a job ran was made by the Sidekiq workers. The image below will help you visualize the process:",[12,242,243,247],{},[164,244],{"alt":245,"src":246},"One queue per worker","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782399025/blog/Content%20Images/sidekiq-2021/One-Queue-Per-Worker.png","\nRepresentation of Sidekiq job routing with one queue per worker",[12,249,250,251,256,257,261],{},"In the image above, each lettered-box represents a queue, and jobs are scheduled into a queue based on their name. Where they execute is up to the workers. As you might imagine, it's entirely possible for Rails to put work into a queue that no worker is configured to pick up. With more than 400 workers that's far too easy, so for GitLab.com we ensured that didn't happen by using ",[28,252,255],{"href":253,"rel":254},"https://docs.gitlab.com/administration/sidekiq/extra_sidekiq_processes/",[],"queue-selector"," expressions for most shards and the ",[28,258,260],{"href":253,"rel":259},[],"negate option"," to define our catchall (default) shard with some scripts to make that easier. It was still a complex process and migrating to Kubernetes added challenges for the final catchall shard as we dealt with the last NFS dependencies and had workloads running in VMs and Kubernetes.",[12,263,264,265,268],{},"With routing rules the decision for which workload should pick up a given job is made when the job is ",[182,266,267],{},"enqueued",". The image below should make it easier to understand this process.",[12,270,271,275],{},[164,272],{"alt":273,"src":274},"One queue per shard Sidekiq job routing with routing rules","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782399025/blog/Content%20Images/sidekiq-2021/One-Queue-Per-Shard.png","\nRepresentation of Sidekiq job routing with routing rules",[12,277,278],{},"Routing rules use the same queue-selector syntax, so the same expressions can still be used to represent shards as before. But because the routing rules are an ordered set of rules applied in the same way for every job no matter where it's scheduled from, we no longer need to use a complex generated \"negate\" expression to define the catchall/default shard.",[12,280,281,282,286,287,290],{},"Instead, all that is required is a final \"default\" rule (",[283,284,285],"code",{},"*",") that matches all remaining jobs and routes them to the catchall shard (we use the ",[283,288,289],{},"default"," queue for that out of convenience). We only need to ensure there is a set of Sidekiq workers listening for each of the resulting small number of simply named queues that we can base on the shard name for obviousness and simplicity. This is much easier to get right and visually verify.",[12,292,26,293,38],{},[28,294,297],{"href":295,"rel":296},"https://docs.gitlab.com/administration/sidekiq/processing_specific_job_classes/",[],"the routing rules syntax and how to configure them",[84,299,301],{"id":300},"how-its-going","How it's going",[12,303,304,305,310],{},"Over the past few months we've been ",[28,306,309],{"href":307,"rel":308},"https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/447",[],"working on migrating GitLab.com to this new arrangement for the catchall shard",". When it came to actually switching to routing rules we took a measured approach and did it in phases.",[12,312,313,314,319],{},"We started by ",[28,315,318],{"href":316,"rel":317},"https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/878/diffs",[],"creating a set of routing rules that recreated our existing shards",", but with the 'nil' target, which tells Rails to keep using the\nper-worker queue. This gave us a base from which we could maintain existing behavior but then start routing to a limited named queues in simple iterative steps.",[12,321,322,323,325,326,331,332,337],{},"From there, we could add new rules immediately before the final catchall rule to the ",[283,324,289],{}," queue, which GitLab doesn't actively use out-of-the-box, but which the catchall shard listens to. First, we ",[28,327,330],{"href":328,"rel":329},"https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/918/diffs",[],"added some rules that don't normally run on GitLab.com",", but which we could use to test (e.g., ",[28,333,336],{"href":334,"rel":335},"https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/chaos/sleep_worker.rb",[],"Chaos::SleepWorker",").",[12,339,340,341,346,347,349,350,38],{},"Next, we moved a feature category with a couple of jobs, first ",[28,342,345],{"href":343,"rel":344},"https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/978/diffs",[],"routing"," them to ",[283,348,289],{}," then we ",[28,351,354],{"href":352,"rel":353},"https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/979/diffs",[],"stopped listening to them in the catchall shard",[12,356,357,358,363,364,369,370,373,374,377],{},"We repeated this pattern of rerouting then not listening in ",[28,359,362],{"href":360,"rel":361},"https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1074#migration-phases",[],"batches"," using ",[28,365,368],{"href":366,"rel":367},"https://docs.gitlab.com/development/feature_categorization/",[],"Feature Categories"," as a relatively simple way to select groups of related workers. After selecting the groups, we built them up from batches with small groups of low-use categories with lots of classes but not many jobs per second to single feature categories at the end. ",[283,371,372],{},"Continuous_integration"," (18% of catchall shard jobs) and ",[283,375,376],{},"integrations and ","source_code_management` are each generating about 30% of catch-all shard jobs. We gained confidence in the queue-handling as this progressed, and gave ourselves plenty of opportunity to gather data and pause if necessary.",[12,379,380],{},"At each stage we stopped listening to several 10s of queues from our catchall Kubernetes deployment, and gradually saw the CPU\nsaturation on Redis drop. After finally shutting down Sidekiq on our (now legacy) virtual machines, we've reached a\nfinal state where at peak times CPU on Redis reaches only around 75%, down from peaks of 95% or higher:",[12,382,383,387],{},[164,384],{"alt":385,"src":386},"Reduced CPU usage in Redis","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782399025/blog/Content%20Images/sidekiq-2021/cpu-then-vs-now.png","\nReduced CPU usage in Redis",[84,389,391],{"id":390},"what-will-we-do-next","What will we do next?",[12,393,394,395,400],{},"First, we need to finish the ",[28,396,399],{"href":397,"rel":398},"https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/469",[],"one-queue-per-shard migration","\nfor all the other shards, aside from catchall. These shards won't have the same level of impact because they run far\nfewer queues than catchall, but will lead to a consistent job routing strategy. In the long term, Redis\nwill eventually become the bottleneck again, and we're going to have to either split Sidekiq into multiple fleets, or change\nto something architecturally different. Multiple fleets has some challenges, but means we can keep using the existing\ntechnologies we have invested time and tooling in (including Omnibus for self-managed deployments). But given the\nbottleneck is still eventually going to be Redis CPU, this might well be the time to look at other job processing\nparadigms.",[12,402,403],{},[182,404,405],{},"In our next blog post, we explain how you can take what we learned about configuring Sidekiq for GitLab.com and apply it to your own large instance of GitLab.",[12,407,408,409,414,415],{},"Cover image by ",[28,410,413],{"href":411,"rel":412},"https://unsplash.com/@z734923105",[],"Jerry Zhang"," on ",[28,416,419],{"href":417,"rel":418},"https://www.unsplash.com",[],"Unsplash",{"title":421,"searchDepth":422,"depth":422,"links":423},"",2,[424,428],{"id":42,"depth":422,"text":43,"children":425},[426],{"id":86,"depth":427,"text":87},3,{"id":120,"depth":422,"text":121,"children":429},[430,431,432,433,434],{"id":124,"depth":427,"text":125},{"id":156,"depth":427,"text":157},{"id":227,"depth":427,"text":228},{"id":300,"depth":427,"text":301},{"id":390,"depth":427,"text":391},"engineering","2021-09-02","Sidekiq is a key part of GitLab, and usually works well out-of-the-box, but sometimes it needs more attention at scale.","md",null,false,"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749667068/Blog/Hero%20Images/sidekiqmountain.jpg",{},true,"/en-us/blog/specialized-sidekiq-configuration-lessons-from-gitlab-dot-com",{"title":5,"description":437,"ogTitle":5,"ogDescription":437,"noIndex":440,"ogImage":441,"ogUrl":446,"ogSiteName":447,"ogType":448,"canonicalUrls":446},"https://about.gitlab.com/blog/specialized-sidekiq-configuration-lessons-from-gitlab-dot-com","https://about.gitlab.com","article","specialized-sidekiq-configuration-lessons-from-gitlab-dot-com","en-us/blog/specialized-sidekiq-configuration-lessons-from-gitlab-dot-com","BlogPost","EuWHYG_VTGUlCTQcRfDXaJCFTlxyiuQ57FWGlmtVyF8",{"logo":454,"freeTrial":459,"sales":464,"login":469,"items":474,"search":801,"minimal":832,"duo":851,"switchNav":860,"pricingDeployment":871},{"config":455},{"href":456,"dataGaName":457,"dataGaLocation":458},"/","gitlab logo","header",{"text":460,"config":461},"Get free trial",{"href":462,"dataGaName":463,"dataGaLocation":458},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":465,"config":466},"Request a demo",{"href":467,"dataGaName":468,"dataGaLocation":458},"/sales/?contact-topic=request-demo","sales",{"text":470,"config":471},"Sign in",{"href":472,"dataGaName":473,"dataGaLocation":458},"https://gitlab.com/users/sign_in/","sign in",[475,504,604,609,723,779],{"text":476,"config":477,"menu":479},"Platform",{"dataNavLevelOne":478},"platform",{"type":480,"columns":481},"cards",[482,488,496],{"title":476,"description":483,"link":484},"The intelligent orchestration platform for DevSecOps",{"text":485,"config":486},"Explore our Platform",{"href":487,"dataGaName":478,"dataGaLocation":458},"/platform/",{"title":489,"description":490,"link":491},"GitLab Duo Agent Platform","Agentic AI for the entire software lifecycle",{"text":492,"config":493},"Meet GitLab Duo",{"href":494,"dataGaName":495,"dataGaLocation":458},"/gitlab-duo-agent-platform/","gitlab duo agent platform",{"title":497,"description":498,"link":499},"Why GitLab","See the top reasons enterprises choose GitLab",{"text":500,"config":501},"Learn more",{"href":502,"dataGaName":503,"dataGaLocation":458},"/why-gitlab/","why gitlab",{"text":505,"left":443,"config":506,"menu":508},"Product",{"dataNavLevelOne":507},"solutions",{"type":509,"link":510,"columns":514,"feature":583},"lists",{"text":511,"config":512},"View all Solutions",{"href":513,"dataGaName":507,"dataGaLocation":458},"/solutions/",[515,539,562],{"title":516,"description":517,"link":518,"items":523},"Automation","CI/CD and automation to accelerate deployment",{"config":519},{"icon":520,"href":521,"dataGaName":522,"dataGaLocation":458},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[524,528,531,535],{"text":525,"config":526},"CI/CD",{"href":527,"dataGaLocation":458,"dataGaName":525},"/solutions/continuous-integration/",{"text":489,"config":529},{"href":494,"dataGaLocation":458,"dataGaName":530},"gitlab duo agent platform - product menu",{"text":532,"config":533},"Source Code Management",{"href":534,"dataGaLocation":458,"dataGaName":532},"/solutions/source-code-management/",{"text":536,"config":537},"Automated Software Delivery",{"href":521,"dataGaLocation":458,"dataGaName":538},"Automated software delivery",{"title":540,"description":541,"link":542,"items":547},"Security","Deliver code faster without compromising security",{"config":543},{"href":544,"dataGaName":545,"dataGaLocation":458,"icon":546},"/solutions/application-security-testing/","security and compliance","ShieldCheckLight",[548,552,557],{"text":549,"config":550},"Application Security Testing",{"href":544,"dataGaName":551,"dataGaLocation":458},"Application security testing",{"text":553,"config":554},"Software Supply Chain Security",{"href":555,"dataGaLocation":458,"dataGaName":556},"/solutions/supply-chain/","Software supply chain security",{"text":558,"config":559},"Software Compliance",{"href":560,"dataGaName":561,"dataGaLocation":458},"/solutions/software-compliance/","software compliance",{"title":563,"link":564,"items":569},"Measurement",{"config":565},{"icon":566,"href":567,"dataGaName":568,"dataGaLocation":458},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[570,574,578],{"text":571,"config":572},"Visibility & Measurement",{"href":567,"dataGaLocation":458,"dataGaName":573},"Visibility and Measurement",{"text":575,"config":576},"Value Stream Management",{"href":577,"dataGaLocation":458,"dataGaName":575},"/solutions/value-stream-management/",{"text":579,"config":580},"Analytics & Insights",{"href":581,"dataGaLocation":458,"dataGaName":582},"/solutions/analytics-and-insights/","Analytics and insights",{"title":584,"type":509,"items":585},"GitLab for",[586,592,598],{"text":587,"config":588},"Enterprise",{"icon":589,"href":590,"dataGaLocation":458,"dataGaName":591},"Building","/enterprise/","enterprise",{"text":593,"config":594},"Small Business",{"icon":595,"href":596,"dataGaLocation":458,"dataGaName":597},"Work","/small-business/","small business",{"text":599,"config":600},"Public Sector",{"icon":601,"href":602,"dataGaLocation":458,"dataGaName":603},"Organization","/solutions/public-sector/","public sector",{"text":605,"config":606},"Pricing",{"href":607,"dataGaName":608,"dataGaLocation":458,"dataNavLevelOne":608},"/pricing/","pricing",{"text":610,"config":611,"menu":613},"Resources",{"dataNavLevelOne":612},"resources",{"type":509,"link":614,"columns":618,"feature":712},{"text":615,"config":616},"View all resources",{"href":617,"dataGaName":612,"dataGaLocation":458},"/resources/",[619,652,679],{"title":620,"items":621},"Getting started",[622,627,632,637,642,647],{"text":623,"config":624},"Install",{"href":625,"dataGaName":626,"dataGaLocation":458},"/install/","install",{"text":628,"config":629},"Quick start guides",{"href":630,"dataGaName":631,"dataGaLocation":458},"/get-started/","quick setup checklists",{"text":633,"config":634},"Learn",{"href":635,"dataGaLocation":458,"dataGaName":636},"https://university.gitlab.com/","learn",{"text":638,"config":639},"Product documentation",{"href":640,"dataGaName":641,"dataGaLocation":458},"https://docs.gitlab.com/","product documentation",{"text":643,"config":644},"Best practice videos",{"href":645,"dataGaName":646,"dataGaLocation":458},"/getting-started-videos/","best practice videos",{"text":648,"config":649},"Integrations",{"href":650,"dataGaName":651,"dataGaLocation":458},"/integrations/","integrations",{"title":653,"items":654},"Discover",[655,660,665,670,674],{"text":656,"config":657},"Customer success stories",{"href":658,"dataGaName":659,"dataGaLocation":458},"/customers/","customer success stories",{"text":661,"config":662},"Blog",{"href":663,"dataGaName":664,"dataGaLocation":458},"/blog/","blog",{"text":666,"config":667},"Demo Hub",{"href":668,"dataGaName":669,"dataGaLocation":458},"/demo-hub/","demo hub",{"text":671,"config":672},"The Source",{"href":673,"dataGaName":664,"dataGaLocation":458},"/the-source/",{"text":675,"config":676},"Remote",{"href":677,"dataGaName":678,"dataGaLocation":458},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"title":680,"items":681},"Connect",[682,687,692,697,702,707],{"text":683,"config":684},"GitLab Services",{"href":685,"dataGaName":686,"dataGaLocation":458},"/services/","services",{"text":688,"config":689},"Contribute",{"href":690,"dataGaName":691,"dataGaLocation":458},"https://contributors.gitlab.com","contribute",{"text":693,"config":694},"Community",{"href":695,"dataGaName":696,"dataGaLocation":458},"/community/","community",{"text":698,"config":699},"Forum",{"href":700,"dataGaName":701,"dataGaLocation":458},"https://forum.gitlab.com/","forum",{"text":703,"config":704},"Events",{"href":705,"dataGaName":706,"dataGaLocation":458},"/events/","events",{"text":708,"config":709},"Partners",{"href":710,"dataGaName":711,"dataGaLocation":458},"/partners/","partners",{"config":713,"title":716,"text":717,"link":718},{"background":714,"textColor":715},"url('https://res.cloudinary.com/about-gitlab-com/image/upload/v1777322348/qpq8yrgn8knii57omj0c.png')","#000","What’s new in GitLab","Stay updated with our latest features and improvements.",{"text":719,"config":720},"Read the latest",{"href":721,"dataGaName":722,"dataGaLocation":458},"/whats-new/","whats new",{"text":724,"config":725,"menu":727},"Company",{"dataNavLevelOne":726},"company",{"type":509,"columns":728},[729],{"items":730},[731,736,742,744,749,754,759,764,769,774],{"text":732,"config":733},"About",{"href":734,"dataGaName":735,"dataGaLocation":458},"/company/","about",{"text":737,"config":738,"footerGa":741},"Jobs",{"href":739,"dataGaName":740,"dataGaLocation":458},"/jobs/","jobs",{"dataGaName":740},{"text":703,"config":743},{"href":705,"dataGaName":706,"dataGaLocation":458},{"text":745,"config":746},"Leadership",{"href":747,"dataGaName":748,"dataGaLocation":458},"/company/team/e-group/","leadership",{"text":750,"config":751},"Handbook",{"href":752,"dataGaName":753,"dataGaLocation":458},"https://handbook.gitlab.com/","handbook",{"text":755,"config":756},"Investor relations",{"href":757,"dataGaName":758,"dataGaLocation":458},"https://ir.gitlab.com/overview/default.aspx","investor relations",{"text":760,"config":761},"Trust Center",{"href":762,"dataGaName":763,"dataGaLocation":458},"/security/","trust center",{"text":765,"config":766},"AI Transparency Center",{"href":767,"dataGaName":768,"dataGaLocation":458},"/ai-transparency-center/","ai transparency center",{"text":770,"config":771},"Newsletter",{"href":772,"dataGaName":773,"dataGaLocation":458},"/company/contact/#contact-forms","newsletter",{"text":775,"config":776},"Press",{"href":777,"dataGaName":778,"dataGaLocation":458},"/press/","press",{"text":780,"config":781,"menu":782},"Contact us",{"dataNavLevelOne":726},{"type":509,"columns":783},[784],{"items":785},[786,791,796],{"text":787,"config":788},"Talk to sales",{"href":789,"dataGaName":790,"dataGaLocation":458},"/sales/","talk to sales",{"text":792,"config":793},"Support portal",{"href":794,"dataGaName":795,"dataGaLocation":458},"https://support.gitlab.com/hc/en-us","support portal",{"text":797,"config":798},"Customer portal",{"href":799,"dataGaName":800,"dataGaLocation":458},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":802,"login":803,"suggestions":810},"Close",{"text":804,"link":805},"To search repositories and projects, login to",{"text":806,"config":807},"gitlab.com",{"href":472,"dataGaName":808,"dataGaLocation":809},"search login","search",{"text":811,"default":812},"Suggestions",[813,815,819,821,825,829],{"text":489,"config":814},{"href":494,"dataGaName":489,"dataGaLocation":809},{"text":816,"config":817},"Code Suggestions (AI)",{"href":818,"dataGaName":816,"dataGaLocation":809},"/solutions/code-suggestions/",{"text":525,"config":820},{"href":527,"dataGaName":525,"dataGaLocation":809},{"text":822,"config":823},"GitLab on AWS",{"href":824,"dataGaName":822,"dataGaLocation":809},"/partners/technology-partners/aws/",{"text":826,"config":827},"GitLab on Google Cloud",{"href":828,"dataGaName":826,"dataGaLocation":809},"/partners/technology-partners/google-cloud-platform/",{"text":830,"config":831},"Why GitLab?",{"href":502,"dataGaName":830,"dataGaLocation":809},{"freeTrial":833,"mobileIcon":838,"desktopIcon":843,"secondaryButton":846},{"text":834,"config":835},"Start free trial",{"href":836,"dataGaName":463,"dataGaLocation":837},"https://gitlab.com/-/trials/new/","nav",{"altText":839,"config":840},"Gitlab Icon",{"src":841,"dataGaName":842,"dataGaLocation":837},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":839,"config":844},{"src":845,"dataGaName":842,"dataGaLocation":837},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":847,"config":848},"Get Started",{"href":849,"dataGaName":850,"dataGaLocation":837},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/get-started/","get started",{"freeTrial":852,"mobileIcon":856,"desktopIcon":858},{"text":853,"config":854},"Learn more about GitLab Duo",{"href":494,"dataGaName":855,"dataGaLocation":837},"gitlab duo",{"altText":839,"config":857},{"src":841,"dataGaName":842,"dataGaLocation":837},{"altText":839,"config":859},{"src":845,"dataGaName":842,"dataGaLocation":837},{"button":861,"mobileIcon":866,"desktopIcon":868},{"text":862,"config":863},"/switch",{"href":864,"dataGaName":865,"dataGaLocation":837},"#contact","switch",{"altText":839,"config":867},{"src":841,"dataGaName":842,"dataGaLocation":837},{"altText":839,"config":869},{"src":870,"dataGaName":842,"dataGaLocation":837},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1773335277/ohhpiuoxoldryzrnhfrh.png",{"freeTrial":872,"mobileIcon":877,"desktopIcon":879},{"text":873,"config":874},"Back to pricing",{"href":607,"dataGaName":875,"dataGaLocation":837,"icon":876},"back to pricing","GoBack",{"altText":839,"config":878},{"src":841,"dataGaName":842,"dataGaLocation":837},{"altText":839,"config":880},{"src":845,"dataGaName":842,"dataGaLocation":837},{"title":882,"titleMobile":883,"button":884,"config":889},"Duo Agent Platform delivers 400% ROI, per new Forrester Consulting study.","400% ROI: Forrester TEI for GitLab Duo",{"text":500,"config":885},{"href":886,"dataGaName":887,"dataGaLocation":888},"https://about.gitlab.com/blog/gitlab-duo-agent-platform-delivers-400-percent-roi/","forrester-tei-dap-banner","global-banner",{"layout":890,"disabled":440},"release",{"data":892},{"text":893,"source":894,"edit":900,"contribute":905,"config":910,"items":915,"minimal":1125},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":895,"config":896},"View page source",{"href":897,"dataGaName":898,"dataGaLocation":899},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":901,"config":902},"Edit this page",{"href":903,"dataGaName":904,"dataGaLocation":899},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":906,"config":907},"Please contribute",{"href":908,"dataGaName":909,"dataGaLocation":899},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":911,"facebook":912,"youtube":913,"linkedin":914},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[916,963,1017,1061,1093],{"title":605,"links":917,"subMenu":932},[918,922,927],{"text":919,"config":920},"View plans",{"href":607,"dataGaName":921,"dataGaLocation":899},"view plans",{"text":923,"config":924},"Why Premium?",{"href":925,"dataGaName":926,"dataGaLocation":899},"/pricing/premium/","why premium",{"text":928,"config":929},"Why Ultimate?",{"href":930,"dataGaName":931,"dataGaLocation":899},"/pricing/ultimate/","why ultimate",[933],{"title":934,"links":935},"Contact Us",[936,939,941,943,948,953,958],{"text":937,"config":938},"Contact sales",{"href":789,"dataGaName":468,"dataGaLocation":899},{"text":792,"config":940},{"href":794,"dataGaName":795,"dataGaLocation":899},{"text":797,"config":942},{"href":799,"dataGaName":800,"dataGaLocation":899},{"text":944,"config":945},"Status",{"href":946,"dataGaName":947,"dataGaLocation":899},"https://status.gitlab.com/","status",{"text":949,"config":950},"Terms of use",{"href":951,"dataGaName":952,"dataGaLocation":899},"/terms/","terms of use",{"text":954,"config":955},"Privacy statement",{"href":956,"dataGaName":957,"dataGaLocation":899},"/privacy/","privacy statement",{"text":959,"config":960},"Cookie preferences",{"dataGaName":961,"dataGaLocation":899,"id":962,"isOneTrustButton":443},"cookie preferences","ot-sdk-btn",{"title":505,"links":964,"subMenu":973},[965,969],{"text":966,"config":967},"DevSecOps platform",{"href":487,"dataGaName":968,"dataGaLocation":899},"devsecops platform",{"text":970,"config":971},"AI-Assisted Development",{"href":494,"dataGaName":972,"dataGaLocation":899},"ai-assisted development",[974],{"title":975,"links":976},"Topics",[977,982,987,992,997,1002,1007,1012],{"text":978,"config":979},"CICD",{"href":980,"dataGaName":981,"dataGaLocation":899},"/topics/ci-cd/","cicd",{"text":983,"config":984},"GitOps",{"href":985,"dataGaName":986,"dataGaLocation":899},"/topics/gitops/","gitops",{"text":988,"config":989},"DevOps",{"href":990,"dataGaName":991,"dataGaLocation":899},"/topics/devops/","devops",{"text":993,"config":994},"Version Control",{"href":995,"dataGaName":996,"dataGaLocation":899},"/topics/version-control/","version control",{"text":998,"config":999},"DevSecOps",{"href":1000,"dataGaName":1001,"dataGaLocation":899},"/topics/devsecops/","devsecops",{"text":1003,"config":1004},"Cloud Native",{"href":1005,"dataGaName":1006,"dataGaLocation":899},"/topics/cloud-native/","cloud native",{"text":1008,"config":1009},"AI for Coding",{"href":1010,"dataGaName":1011,"dataGaLocation":899},"/topics/devops/ai-for-coding/","ai for coding",{"text":1013,"config":1014},"Agentic AI",{"href":1015,"dataGaName":1016,"dataGaLocation":899},"/topics/agentic-ai/","agentic ai",{"title":1018,"links":1019},"Solutions",[1020,1022,1024,1029,1033,1036,1040,1043,1045,1048,1051,1056],{"text":549,"config":1021},{"href":544,"dataGaName":549,"dataGaLocation":899},{"text":538,"config":1023},{"href":521,"dataGaName":522,"dataGaLocation":899},{"text":1025,"config":1026},"Agile development",{"href":1027,"dataGaName":1028,"dataGaLocation":899},"/solutions/agile-delivery/","agile delivery",{"text":1030,"config":1031},"SCM",{"href":534,"dataGaName":1032,"dataGaLocation":899},"source code management",{"text":978,"config":1034},{"href":527,"dataGaName":1035,"dataGaLocation":899},"continuous integration & delivery",{"text":1037,"config":1038},"Value stream management",{"href":577,"dataGaName":1039,"dataGaLocation":899},"value stream management",{"text":983,"config":1041},{"href":1042,"dataGaName":986,"dataGaLocation":899},"/solutions/gitops/",{"text":587,"config":1044},{"href":590,"dataGaName":591,"dataGaLocation":899},{"text":1046,"config":1047},"Small business",{"href":596,"dataGaName":597,"dataGaLocation":899},{"text":1049,"config":1050},"Public sector",{"href":602,"dataGaName":603,"dataGaLocation":899},{"text":1052,"config":1053},"Education",{"href":1054,"dataGaName":1055,"dataGaLocation":899},"/solutions/education/","education",{"text":1057,"config":1058},"Financial services",{"href":1059,"dataGaName":1060,"dataGaLocation":899},"/solutions/finance/","financial services",{"title":610,"links":1062},[1063,1065,1067,1069,1072,1074,1077,1079,1081,1083,1085,1087,1089,1091],{"text":623,"config":1064},{"href":625,"dataGaName":626,"dataGaLocation":899},{"text":628,"config":1066},{"href":630,"dataGaName":631,"dataGaLocation":899},{"text":633,"config":1068},{"href":635,"dataGaName":636,"dataGaLocation":899},{"text":638,"config":1070},{"href":640,"dataGaName":1071,"dataGaLocation":899},"docs",{"text":661,"config":1073},{"href":663,"dataGaName":664,"dataGaLocation":899},{"text":1075,"config":1076},"What's new",{"href":721,"dataGaName":722,"dataGaLocation":899},{"text":656,"config":1078},{"href":658,"dataGaName":659,"dataGaLocation":899},{"text":675,"config":1080},{"href":677,"dataGaName":678,"dataGaLocation":899},{"text":683,"config":1082},{"href":685,"dataGaName":686,"dataGaLocation":899},{"text":688,"config":1084},{"href":690,"dataGaName":691,"dataGaLocation":899},{"text":693,"config":1086},{"href":695,"dataGaName":696,"dataGaLocation":899},{"text":698,"config":1088},{"href":700,"dataGaName":701,"dataGaLocation":899},{"text":703,"config":1090},{"href":705,"dataGaName":706,"dataGaLocation":899},{"text":708,"config":1092},{"href":710,"dataGaName":711,"dataGaLocation":899},{"title":724,"links":1094},[1095,1097,1099,1101,1103,1105,1109,1114,1116,1118,1120],{"text":732,"config":1096},{"href":734,"dataGaName":726,"dataGaLocation":899},{"text":737,"config":1098},{"href":739,"dataGaName":740,"dataGaLocation":899},{"text":745,"config":1100},{"href":747,"dataGaName":748,"dataGaLocation":899},{"text":750,"config":1102},{"href":752,"dataGaName":753,"dataGaLocation":899},{"text":755,"config":1104},{"href":757,"dataGaName":758,"dataGaLocation":899},{"text":1106,"config":1107},"Sustainability",{"href":1108,"dataGaName":1106,"dataGaLocation":899},"/sustainability/",{"text":1110,"config":1111},"Diversity, inclusion and belonging (DIB)",{"href":1112,"dataGaName":1113,"dataGaLocation":899},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":760,"config":1115},{"href":762,"dataGaName":763,"dataGaLocation":899},{"text":770,"config":1117},{"href":772,"dataGaName":773,"dataGaLocation":899},{"text":775,"config":1119},{"href":777,"dataGaName":778,"dataGaLocation":899},{"text":1121,"config":1122},"Modern Slavery Transparency Statement",{"href":1123,"dataGaName":1124,"dataGaLocation":899},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"items":1126},[1127,1130,1133],{"text":1128,"config":1129},"Terms",{"href":951,"dataGaName":952,"dataGaLocation":899},{"text":1131,"config":1132},"Cookies",{"dataGaName":961,"dataGaLocation":899,"id":962,"isOneTrustButton":443},{"text":1134,"config":1135},"Privacy",{"href":956,"dataGaName":957,"dataGaLocation":899},[1137],{"id":1138,"title":7,"body":439,"config":1139,"content":1141,"description":439,"extension":1145,"meta":1146,"navigation":443,"path":1147,"seo":1148,"stem":1149,"__hash__":1150},"blogAuthors/en-us/blog/authors/craig-miskell.yml",{"template":1140},"BlogAuthor",{"name":7,"config":1142},{"headshot":1143,"ctfId":1144},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749667372/Blog/Author%20Headshots/cmiskell-headshot.jpg","cmiskell","yml",{},"/en-us/blog/authors/craig-miskell",{},"en-us/blog/authors/craig-miskell","VPOHmA7ISNMfEE3Y0SkxSJD1DRgIqQVOLz9bf8UCnh8",[1152,1161,1169],{"title":1153,"description":1154,"heroImage":1155,"category":435,"date":1156,"authors":1157,"slug":1160,"externalUrl":439},"Confidential AI for GitLab Self-Hosted","Give developers AI coding agents in GitLab Duo without source code leaving a hardware-encrypted boundary — no GPUs needed.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1773866173/vte9qh8rriznvyclhkes.png","2026-08-06",[1158,1159],"Mathias Ewald","Martin Paloncy, Edgeless Systems","confidential-ai-for-gitlab-self-hosted",{"title":1162,"description":1163,"heroImage":1164,"category":435,"date":1165,"authors":1166,"slug":1168,"externalUrl":439},"Green DevOps: Why carbon measurement belongs in your CI/CD pipeline","CI/CD pipelines have a hidden carbon cost. Here's why measuring it matters, and how you can get started with Eco CI and Carmen in GitLab.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1765809212/noh0mdfn9o94ry9ykura.png","2026-07-09",[1167],"Lysanne Pinto","green-devops-carbon-measurement-cicd-pipeline",{"title":1170,"description":1171,"heroImage":1172,"category":435,"date":1173,"authors":1174,"slug":1176,"externalUrl":439},"How to build CI/CD observability at scale","This practical guide to GitLab pipeline analytics helps self-managed users gain operational insights using Prometheus and Grafana.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1774465167/n5hlvrsrheadeccyr1oz.png","2026-04-28",[1175],"Paul Meresanu","how-to-build-ci-cd-observability-at-scale",{"promotions":1178},[1179,1193,1205,1217],{"id":1180,"categories":1181,"header":1183,"text":1184,"button":1185,"image":1190},"ai-modernization",[1182],"ai","Is AI achieving its promise at scale?","Quiz will take 5 minutes or less",{"text":1186,"config":1187},"Get your AI maturity score",{"href":1188,"dataGaName":1189,"dataGaLocation":664},"/assessments/ai-modernization-assessment/","modernization assessment",{"config":1191},{"src":1192},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138786/qix0m7kwnd8x2fh1zq49.png",{"id":1194,"categories":1195,"header":1197,"text":1184,"button":1198,"image":1202},"devops-modernization",[1196,1001],"product","Are you just managing tools or shipping innovation?",{"text":1199,"config":1200},"Get your DevOps maturity score",{"href":1201,"dataGaName":1189,"dataGaLocation":664},"/assessments/devops-modernization-assessment/",{"config":1203},{"src":1204},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138785/eg818fmakweyuznttgid.png",{"id":1206,"categories":1207,"header":1209,"text":1184,"button":1210,"image":1214},"security-modernization",[1208],"security","Are you trading speed for security?",{"text":1211,"config":1212},"Get your security maturity score",{"href":1213,"dataGaName":1189,"dataGaLocation":664},"/assessments/security-modernization-assessment/",{"config":1215},{"src":1216},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1772138786/p4pbqd9nnjejg5ds6mdk.png",{"id":1218,"paths":1219,"header":1222,"text":1223,"button":1224,"image":1229},"github-azure-migration",[1220,1221],"migration-from-azure-devops-to-gitlab","integrating-azure-devops-scm-and-gitlab","Is your team ready for GitHub's Azure move?","GitHub is already rebuilding around Azure. Find out what it means for you.",{"text":1225,"config":1226},"See how GitLab compares to GitHub",{"href":1227,"dataGaName":1228,"dataGaLocation":664},"/compare/gitlab-vs-github/github-azure-migration/","github azure migration",{"config":1230},{"src":1204},{"header":1232,"blurb":1233,"button":1234,"secondaryButton":1239},"Start building faster today","See what your team can do with the intelligent orchestration platform for DevSecOps.\n",{"text":1235,"config":1236},"Get your free trial",{"href":1237,"dataGaName":463,"dataGaLocation":1238},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":937,"config":1240},{"href":789,"dataGaName":468,"dataGaLocation":1238},1786803752764]