Le September 30, 2026, par Benjamin Boudier.
Local and open-source AI: a strategic lever for businesses
Local and open-source hosting of artificial intelligence models gives companies full control over their data, guarantees maximum privacy, and reduces long-term costs compared to public cloud solutions. This approach enables the integration of AI capabilities directly into your existing infrastructure, without relying on external providers or exposing your sensitive data to third-party APIs.
In a context where data sovereignty is becoming critical for businesses, local AI represents a strategic solution. In addition to data privacy and lower API call costs, local AI offers flexibility in model selection, avoids cloud vendor lock-in, and simplifies compliance with regulatory requirements.
1. What is open-source AI, and why does it matter for businesses?
Open-source AI refers to artificial intelligence models whose code, weights, and sometimes training data are publicly accessible. These models are generally distributed under permissive licenses, such as Apache 2.0 or MIT, which allow commercial use, modification, and distribution.
The key stakes for businesses :
- Data sovereignty : Unlike cloud APIs that send your data to a third party, local AI processes everything internally.
- Regulatory Compliance : The GDPR and other regulations sometimes require that sensitive data not leave the organization. On-premises AI perfectly meets this requirement.
- Cost reduction : Cloud APIs are often billed on a pay-as-you-go basis or based on the number of tokens. Over the long term, the initial hardware investment pays for itself because there are no recurring costs.
- Technical flexibility : You choose the models that best suit your specific needs, and you can fine-tune them and integrate them directly into your architecture.
- Independance : No reliance on a single provider. You have full control over your infrastructure and can migrate or scale without any restrictions.
2. Understanding models : how they work, types, and choosing the right one based on your needs
A large language model (LLM) is an advanced artificial intelligence system trained on vast amounts of text data to understand and generate natural human language. It essentially functions as a statistical prediction engine that anticipates the most likely word or sequence of words to produce coherent responses, text, or code.
Where to find these models?
Repositories like Hugging Face have become the go-to source for discovering and downloading open-source models. They contain thousands of models with clearly stated licenses. For commercial use, opt for models licensed under the Apache 2.0 or MIT licenses, which are the most permissive.
Key concepts
The tokens
A token represents a part of a text (word, subword, or character). Models process text in tokens. For example, “artificial intelligence” can be divided into 3–5 tokens, depending on the model. The context length (the maximum number of tokens processed simultaneously) ranges from 32K to 1M+ tokens for recent models.
The quantization
Quantization is a compression technique that reduces the precision of a model's weights to decrease its size and memory footprint. Common formats in the GGUF format (used by Ollama and llama.cpp) include :
| Level | Description | VRAM saved | Quality |
|---|---|---|---|
| F16 (Full 16-bit) | Original accuracy | 0% | Maximal |
| Q8_0 | High level of accuracy | ~50% | Excellent |
| Q6_K | Balance quality/size | ~60% | Very good |
| Q4_K_M | Recommended standard | ~72% | Optimal |
| Q3_K_M / Q2_K | Aggressive compression | ~75-80% | Degraded |
The Q4_K_M setting is considered the ideal choice for most users: it reduces VRAM usage by 72% while maintaining a quality very close to that of the original model.
Architecture
- Dense : All parameters are active for each prediction. Simpler, but more resource-intensive.
- MoE (Mixture of Experts) : Only a subset of the parameters (the “experts”) is active for each token. This is more efficient and allows for very large models while still performing well on consumer-grade hardware.
Selecting models and equipment based on needs
| Usage | Equipment | Models | Case |
|---|---|---|---|
| Limited |
|
Q4_K_M:
|
|
| Professionnal |
|
Q4_K_M:
|
|
| Advanced |
|
|
|
Note: The website canirun.ai allows you to check whether your hardware is compatible with different models. It automatically detects your GPU or Mac, then ranks open-source models based on their ability to run smoothly on your system.
3. Getting started with Ollama : set up in just a few minutes
Ollama is an open-source tool that simplifies the local execution of language models. It handles the downloading, execution, and optimization of models with minimal configuration.
Docker configuration
Before continuing, you'll also need to install Docker on your machine by following the official documentation.
docker-compose.yml (CPU-only)
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
ports:
- "11434:11434"
volumes:
- ollama_data:/root/.ollama
volumes:
ollama_data:
docker-compose.gpu.yml (avec GPU NVIDIA)
services:
ollama:
image: ollama/ollama:latest
container_name: ollama-gpu
ports:
- "11434:11434"
volumes:
- ollama_data:/root/.ollama
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
volumes:
ollama_data:
$ docker-compose up -d
Installing a model
To view the models available for Ollama, look at the online store.
This allows you to specify which templates to install locally:
$ docker exec -it ollama ollama pull codellama:7b
$ docker exec -it ollama ollama run codellama:7b
To view the list of installed models:
$ docker exec -it ollama ollama list
Integration wuth your application
Ollama exposes a simple REST API that can be called from any programming language. For example:
$ curl -X POST http://127.0.0.1:11434/api/chat \
-H "Content-Type: application/json" \
-d '{
"model": "codellama:7b",
"messages": [{"role": "user", "content": "Hello"}],
"stream": false
}
Configuration for production
For a production environment and a multi-user scenario, it’s best to use tools like vLLM, which offers up to 10 times the throughput and significantly lower latency, and can efficiently handle thousands of concurrent requests—unlike Ollama.
Complementary resources
Conclusion
On-premises, open-source AI represents a strategic opportunity for companies that want to take control of their data and AI costs. With tools like Ollama and ready-to-use Docker stacks, getting started in this field is now accessible even to teams with limited expertise in ML infrastructure.
The choice of model and hardware depends on your specific needs and budget. Start with a simple setup, evaluate its performance, and then scale up as needed. The key is to start experimenting to understand how AI can transform your business processes.