💡Running a Local LLM for DevSecOps: Exploring Practical AI on a Local Machine
Why I Started Exploring Local AI
Artificial Intelligence has become an integral part of modern software development. From code generation to intelligent assistants, Large Language Models (LLMs) are changing how engineers work.
However, working in DevSecOps presents a unique challenge.
Much of our day-to-day work involves sensitive information:
Infrastructure-as-Code (Terraform, ARM templates)
Kubernetes manifests
Azure DevOps pipelines
Security scan reports
Internal runbooks and operational procedures
Incident postmortems
Compliance documentation
Sending this information to external AI services is often not an option due to organisational policies, regulatory requirements, or security concerns.
This led me to an interesting question:
Can we leverage AI capabilities entirely within our own infrastructure?
The answer was yes.
Running an LLM Locally
I decided to experiment with running a Large Language Model locally on a standard enterprise Linux server.
My environment was fairly modest:
Operating System : RHEL 9
CPU : 8 vCPUs
Memory : 31 GB RAM
GPU : None
After evaluating several open-source models, I chose:
Qwen2.5-7B-Instruct
using:
llama.cpp
The setup process was surprisingly straightforward.
A simple shell script was enough to launch an interactive AI assistant directly from the terminal:
#!/bin/bash
MODEL="$HOME/models/Qwen2.5-7B-Instruct-Q4_K_M.gguf"
LLAMA="$HOME/llama/llama.cpp/build/bin/llama-cli"
"$LLAMA" \
-m "$MODEL" \
-t "$(nproc)" \
-c 8192 \
-cnv
Executing:
./run-llm.sh
provided a fully local conversational interface.
No cloud APIs.
No internet connectivity requirements after setup.
No external data sharing.
Practical Applications in DevSecOps
Once the model was running, I started exploring how it could assist with daily DevSecOps activities.
The opportunities became immediately apparent.
Infrastructure-as-Code Reviews
Infrastructure definitions can become complex over time.
Local LLMs can assist engineers by reviewing:
Terraform modules
Kubernetes manifests
Helm charts
Azure Resource Manager templates
Example prompts:
Review this Terraform configuration for security best practices.
Identify potential risks in this Kubernetes deployment manifest.
Suggest improvements to this Helm chart.
The model acts as an additional pair of eyes during development.
Security Findings Interpretation
Security tools generate extensive reports that often require manual analysis.
Examples include:
Trivy vulnerability scans
Checkov policy violations
Dependency scan reports
Container image assessments
A local LLM can help by:
Summarising findings
Explaining vulnerabilities in simpler language
Suggesting remediation approaches
Highlighting critical items requiring immediate attention
Instead of reading hundreds of lines of output, engineers receive concise and actionable summaries.
CI/CD Pipeline Reviews
Modern deployment pipelines contain significant business logic and security controls.
Local AI assistants can review:
Azure DevOps YAML pipelines
GitHub Actions workflows
Jenkins pipelines
Questions such as:
Are there any security concerns in this pipeline?
Are secrets being handled appropriately?
What best practices are missing?
can significantly improve pipeline quality.
The Future: Beyond Conversations
Running a local LLM is only the beginning.
These models can eventually be integrated into Retrieval-Augmented Generation (RAG) solutions that connect AI assistants with internal documentation and organisational knowledge bases.
This enables systems capable of answering questions based on company-specific information while keeping sensitive data within organisational boundaries.
The local LLM becomes the intelligence layer powering these experiences.
Why This Matters
For years, organisations faced a difficult trade-off:
Use AI capabilities
OR
Protect sensitive enterprise information
Local LLMs significantly reduce this dilemma.
They offer a path towards:
Privacy-preserving AI adoption
Greater control over organisational data
Reduced dependence on external services
Lower operational costs for routine AI-assisted workflows
Most importantly, they empower engineering teams to experiment responsibly.
Example Session
Loading Qwen 2.5 7B...
Type /exit or press Ctrl+C to quit.
Loading model...
▄▄ ▄▄
██ ██
██ ██ ▀▀█▄ ███▄███▄ ▀▀█▄ ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██ ██ ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
██ ██
▀▀ ▀▀
build : b9586-76da2450a
model : Qwen2.5-7B-Instruct-Q4_K_M.gguf
modalities : text
available commands:
/exit or Ctrl+C stop or exit
/regen regenerate the last response
/clear clear the chat history
/read <file> add a text file
/glob <pattern> add text files using globbing pattern
> hi
Hello! How can I assist you today?
[ Prompt: 23.7 t/s | Generation: 7.7 t/s ]
> what is helm ? explain helm chart folder structure
Helm is a package manager for Kubernetes. It simplifies the deployment, scaling, and management of applications on Kubernetes. Helm charts are the packages that Helm uses to install and manage Kubernetes applications.
### Helm Chart Folder Structure
A Helm chart is a directory that contains a structured set of files and subdirectories. The structure of a Helm chart typically includes:
1. `Chart.yaml`
- Contains chart metadata such as name, version, and description.
2. `values.yaml`
- Contains default configuration values that can be overridden during installation.
3. `templates/`
- Stores Kubernetes manifest templates.
4. `charts/`
- Contains chart dependencies.
5. `templates/_helpers.tpl`
- Stores reusable template helpers.
6. `templates/NOTES.txt`
- Displays post-installation instructions.
Blog Reference
The complete setup scripts and examples used in this article are available on GitHub:
https://github.com//local-devsecops-llm
Final Thoughts
What started as an experiment to run an open-source language model on a Linux server evolved into a broader understanding of enterprise AI.
The true value was not simply hosting an LLM locally.
It was recognising the opportunities this creates within DevSecOps and Infrastructure-as-Code practices.
AI does not have to exist outside organisational boundaries.
It can become a trusted assistant operating within them.
For DevSecOps teams managing increasingly complex infrastructures, this represents an exciting opportunity:
To improve productivity, enhance security reviews, accelerate knowledge sharing, and support engineers without compromising control over sensitive information.
For me, this journey has only just begun.
