Code Llama 34B models leave out fill-in-the-middle

Meta has released Code Llama, a family of downloadable coding models built on Llama 2. The August 24 launch includes 7-billion-, 13-billion- and 34-billion-parameter versions, each offered as a general code model, a Python specialist and an instruction-following assistant.

File photo: Meta Platforms Headquarters Menlo Park California, by LPS.1 (2022). CC0 1.0. Cropped and resized; this isn’t a Code Llama launch photograph.

The practical choice goes beyond model size. The launch’s original release instructions and first technical paper describe different capabilities for completing a file, answering a request and handling a large code context. The strongest headline score doesn’t identify one model that does all three best.

Infilling changes the model choice

Infilling means generating code between an existing prefix and suffix, rather than simply continuing text from the cursor. At release, that capability belongs to the 7B and 13B general Code Llama models and their Instruct counterparts. The 34B models and all three Python-specialist models were trained without the infilling objective.

That makes a difference for an editor integration. Moving from a 13B model to the larger 34B version doesn’t preserve every feature. Likewise, a developer working in Python can’t assume that the Python-labelled model supplies fill-in-the-middle completion.

The paper’s benchmark table makes the distinction more concrete. Meta reports 53.7% on HumanEval and 56.2% on MBPP for Code Llama-Python 34B. The general Code Llama 34B records 48.8% and 55.0%, respectively. These are Meta’s pass-at-one results, using greedy decoding, with zero-shot HumanEval and three-shot MBPP prompts. They measure performance on those tests, rather than production-code correctness or editor latency.

For a conversational coding assistant, Instruct is the relevant variant. The general and Python models expect a continuation-style prompt; they aren’t instruction-tuned. The initial repository also requires the prescribed chat formatting for Instruct, so changing model names without changing the prompt protocol can miss the intended behaviour.

A 100,000-token input isn’t a retrieval guarantee

Meta’s long-context fine-tuning used sequences of 16,384 tokens. Its researchers then tested extrapolation beyond that length, reporting stable language-model behaviour around 100,000 tokens and better long-file completion than versions without that training stage.

The same paper reports a decline in synthetic key-retrieval accuracy beyond 16,000 tokens. Even within that length, performance depends on the model and where the required information appears. Accepting a long file and reliably recovering a particular detail from it are separate capabilities.

There’s another qualification in the long-completion experiment: the authors disclose that Long Code Completion (LCC) benchmark data points occur in their code-training data. They resampled validation and test examples to cover longer files more evenly. Those results help describe the experiment, but they don’t establish performance on a wholly unseen customer repository.

Long context also has a deployment cost. The launch implementation allocates attention caches according to configured maximum sequence length and batch size. The README tells operators to set both for their hardware and use case; the advertised context ceiling isn’t a promise that every setting fits on one GPU.

Downloadable weights still have conditions

Code Llama extends the commercial Llama 2 release. Its initial download flow requires licence acceptance and approval before Meta emails a signed download link. The repository supplies a minimal inference example, not a complete editor product.

The linked Llama 2 community licence permits commercial use subject to its conditions. Redistributing the model materials requires a copy of the agreement and the specified attribution notice, and the materials or their outputs can’t be used to improve other large language models except Llama 2 or its derivatives. The README separately warns that generated code may be subject to third-party licences.

The launch model card calls for application-specific safety testing and tuning. Model size, an infilling flag, a context limit and a benchmark score each answer a different deployment question. The release gives developers those choices, rather than removing the need to test their combination.