Llama 3 chat needs more than the right model weights

Meta has released Llama 3 in 8-billion- and 70-billion-parameter sizes, each with a pretrained version and an instruction-tuned version for dialogue. The April 18 announcement puts downloadable text models at the centre of the launch. Longer context, multimodality and models above 400 billion parameters remain future work.

For developers, the practical change reaches beyond choosing a model size. The launch code makes conversation formatting, generation limits and checkpoint layout part of the deployment decision.

File photo: Meta headquarters, LPS.1, 2022. CC0 1.0, via Wikimedia Commons. Cropped and resized; not the Llama 3 launch.

The end of a turn is a separate signal

The launch-day instructions distinguish plain text continuation from chat. Pretrained weights expect a prompt whose answer continues the text. Instruction-tuned weights expect messages with explicit role headers, separators and an end-of-turn marker.

The tokenizer implementation shows another important step: after encoding the conversation, it appends an empty assistant header for the model to complete. It also recognises both the end-of-text token, <|end_of_text|>, and the separate end-of-turn token, <|eot_id|>, as stopping signals.

Those details belong together. A chat integration needs to construct the expected conversation and recognise when the assistant’s reply has finished. Matching only the checkpoint name doesn’t establish that a serving wrapper follows the launch format. This is a reading of the reference implementation, rather than a test of every third-party server.

An 8K window is shared with the answer

Meta’s model card specifies an 8K context window for both sizes. The reference code makes the operational limit more concrete.

In generation.py, total sequence length is capped at the configured maximum. Prompt tokens and newly generated tokens share that space. Increasing the requested answer length therefore can’t create extra room beyond the configured window.

Memory allocation is a separate constraint. The attention implementation allocates key and value caches using both maximum batch size and maximum sequence length, alongside the local attention-head dimensions. Raising those settings increases the cache allocation even before a request fills the available space. The model’s advertised context capacity and an application’s chosen memory budget are different quantities.

The README lists model-parallel settings of one for 8B and eight for 70B. The loader checks that its parallel size matches the checkpoint-file count. That’s the layout expected by this reference path, not an independently established minimum GPU count for every possible implementation. Meta also distributes a Transformers format separately from the native files.

Download access has its own conditions

Meta’s download flow requires licence acceptance and approval; Hugging Face access is also gated. The launch licence permits commercial use subject to conditions. Attribution requirements cover distributions and products or services using the materials; AI models made using them and distributed or made available must have names beginning with “Llama 3”. The licence restricts using the materials or their outputs to improve other large language models, except Llama 3 and its derivatives, and incorporates an acceptable-use policy.

Its additional commercial terms assess the licensee and its affiliates at the April 18 release date: if products or services made available by or for them exceeded 700 million monthly active users in the preceding calendar month, a separate licence from Meta is required before exercising the agreement’s rights. Recipients obtaining the materials as part of an integrated end-user product are exempt from that additional condition, not from the entire agreement.

The launch card describes intended commercial and research use in English, while allowing developers to fine-tune for other languages under the licence and policy. It gives different training-data cutoffs for the two sizes: March 2023 for 8B and December 2023 for 70B. Neither checkpoint becomes a live information service simply because it was released this month.